Distributed Computing Working Group

From R Consortium Wiki
Revision as of 11:11, 18 October 2016 by MichaelLawrence (Talk | contribs)

Jump to: navigation, search

Goals and Purpose

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.


  • Michael Lawrence (Genentech)
  • Indrajit Roy (HP Enterprise)
  • Joe Rickert (Microsoft)
  • Bernd Bischl (LMU)
  • Matt Dowle (H2O)
  • Mario Inchiosa (Microsoft)
  • Michael Kane (Yale)
  • Javier Luraschi (RStudio)
  • Edward Ma (HP Enterprise)
  • Luke Tierney (University of Iowa)
  • Simon Urbanek (AT&T)



  • Introduced Clark who is the intern funded by R Consortium. Clark is a graduate student from UC Davis. He will work on ddR integration with Spark and improving the core ddR API as well such as adding a distributed apply() for matrices, split function, etc.
  • Bernd: Can I play around with ddR now? What backend should I use? How robust is the code?
    • Clark: It's in good enough shape to be played around with. We will continue to improve it. Hopefully the spark integration will be done before the end of my internship in September.
  • Q: Is anyone working on using ddR to make ML scale better.
    • Indrajit: We have kmeans, glm, etc. already in CRAN.
    • Michael Kane: We are working on glmnet and other packages related to algorithm development.
  • Javier gave a demo of sparklyr and sparkapi.
    • Motivation for the pacakage: The SparkR package overrides the dplyr interface. This is an issue for RStudio. SparkR is not a CRAN package which makes it difficult to add changes. dplyr is the most popular tool by RStudio and is broken on SparkR.
    • Sparklyr provides a dplyr interface. It will also support ML like interfaces, such as consuming a ML model.
    • Sparklyr does not currently support any distributed computing features. Instead we can recommend ddR as the distributed computing framework on top of sparkapi. We will put the code in CRAN in a couple of weeks.
    • Simon: Can you talk more about the wrapper/low level API to work with Spark?
      • Javier: The package underneath the cover is called "sparkapi" it is to be used by pacakge builders. "spark_context()" and "invoke()" are the functionality to call scala methods. It does not you to currently run R user defined functions. I am currently working on enabling that feature. Depending upon the interest in using ddR with sparkapi, I can spend more time to make sparkapi feature rich.
    • Indrajit: What versions of Spark are supported
      • Javier: Anything after 1.6
    • Bernd: How do you export data?
      • Javier: We are using all the code from SparkR. So everything in SparkR should continue to work. We don't need to change SparkR. We just need to maintain compatibility.
    • Bernd: What happens when the RDDs are very large?
      • Javier: Spark will spill on disk.
  • Michael Kane: Presented examples that he implemented on ddR.
    • Talked about how the different distributed packages compare to each other in terms of functionality.
    • Michael K. looked at glm and truncated SVD on ddR. Was able to implement irls on ddR by implementing two distributed functions such as "cross". In truncated SVD only needed to overload two different distributed multiplications.
    • Ran these algorithms on the 1000 genome dataset.
    • Overall liked ddR since it was easy to implement the algorithms in the package.
    • New ideas:
      • Trying to separate the data layer from the execution layer
      • Create an API that works on "chunks" (which is similar to the "parts" API in ddR). Would like to add these APIs to ddR.
      • Indrajit: You should be able to get some of the chunk like features by using parts and dmapply. E.g., you can call dmapply to read 10 different files, which correspond 10 chunks now. These are however wrapped as a darray or dframe. But you can continue to work on the individual chunks by using parts(i).


  • Round table introduction
  • (Michael) Goals for the group:
    • Make a common abstraction/interfaces to make it easier to work with distributed data and R
    • Unify the interface
    • Working group will run for a year. Get an API defined, get at least one open source reference implementations
    • not everyone needs to work hands on. We will create smaller groups to focus on those aspects.
    • We tried to get a diverse group of participants
  • Logistics: meet monthly, focus groups may meet more often
  • R Consoritum may be able to figure ways to fund smaller projects that come out of the working group
  • Michael Kane: Should we start with an inventory of what is available and people are using?
    • Michael Lawrence: Yes, we should find the collection of tools as well as the use cases that are common.
    • Joe: I will figure out a wiki space.
  • Javier: Who are the end users? Simon: Common layer needed to get algorithms working. We started from algos and tried to find the minimal common api. One of the goals is to make sure everyone is on the same page and not trying to create his/her own custom interface.
  • Javier: Should we try to get people with more algo expertise?
  • Joe: Simon do you have a stack diagram?
  • Simon: Can we get R Consortium to help write things up and draw things?
  • Next meeting: Javier is going to present SparkR next time.