R Consortium Wiki - User contributions [en]

Distributed Computing Working Group Progress Report 2016

2017-05-01T18:45:03Z

MichaelLawrence:

Authors: Michael Lawrence and Indrajit Roy

== Introduction ==

Data sizes continue to increase, while single core performance has
stagnated. We scale computations by leveraging multiple cores and
machines. Large datasets are expensive to replicate, so we minimize
data movement by moving the computation to the data. Many systems,
such as Hadoop, Spark, and massively parallel processing (MPP)
databases, have emerged to support these strategies, and each exposes
its own unique interface, with little standardization.

Developing and executing an algorithm in the distributed context is a
complex task that requires specific knowledge of and dependency on the
system storing the data. It is also a task orthogonal to the primary
role of a data scientist or statistician: extracting knowledge from
data. The task thus falls to the data analysis environment, which
should mask the complexity behind a familiar interface, maintaining
user productivity. However, it is not always feasible to automatically
determine the optimal strategy for a given problem, so user input is
often beneficial. The environment should only abstract the details to
the extent deemed appropriate by the user.

R needs a standardized, layered and idiomatic abstraction for
computing on distributed data structures. R has many packages that
provide parallelism constructs as well as bridges to distributed
systems such as Hadoop. Unfortunately, each interface has its own
syntax, parallelism techniques, and supported platform(s). As a
consequence, contributors are forced to learn multiple idiosyncratic
interfaces, and to restrict each implementation to a particular
interface, thus limiting the applicability and adoption of their
software and hampering interoperability.

The idea of a unified interface stemmed from a cross-industry workshop
organized at HP Labs in early 2015. The workshop was attended by
different companies, universities, and R-core members. Immediately
after the workshop, Indrajit Roy, Edward Ma, and Michael Lawrence began
designing an abstraction that later became known as the CRAN package
ddR (Distributed Data in R)[1]. It declares a unified API for distributed
computing in R and ensures that R programs written using the API are
portable across different systems, such as Distributed R, Spark, etc.

The ddR package has completed its initial phase of development; the
first release is now on CRAN. Three ddR machine-learning algorithms
are also on CRAN, randomForest.ddR, glm.ddR, and kmeans.ddR. Two
reference backends for ddR have been completed, one for R’s parallel
package, and one for HP Distributed R. Example code and scripts to run
algorithms and code on both of these backends are available in our
public repository at https://github.com/vertica/ddR.

The overarching goal of the ddR project was for it to be a starting
point in a collaborative effort, ultimately leading to a standard API
for working with distributed data in R. We decided that it was
natural for the R Consortium to sponsor the collaboration, as it
should involve both industry and R-core members. To this end, we
established the R Consortium Working Group on Distributed Computing,
with a planned duration of a single year and the following aims:

# Agree on the goal of the group, i.e., we should have a unifying framework for distributed computing. Define success metric.
# Brainstorm on what primitives should be included in the API. We can use ddR’s API of distributed data-structures and dmapply as the starting proposal. Understand relationship with existing packages such as parallel, foreach, etc.
# Explore how ddR like interface will interact with databases. Are there connections or redundancies with dplyr and multiplyr?
# Decide on a reference implementation for the API.
# Decide on whether we should also implement a few ecosystem packages, e.g., distributed algorithms written using the API.

We declared the following milestones:

# Mid-year milestone: Finalize API. Decide who all will help with developing the top-level implementation and backends.
# End-year milestone: Summary report and reference implementation. Socialize the final package.

This report outlines the progress we have made on the above goals and
milestones, and how we plan to continue progress in the second half of
the working group term.

== Results and Current Status ==

The working group has achieved the first goal by agreeing that we
should aim for a unifying distributed computing abstraction, and we
have treated ddR as an informal API proposal.

We have discussed many of the issues related to the second goal,
deciding which primitives should be part of the API. We aim for the
API to support three shapes of data --- lists, arrays and data frames
--- and to enable the loading and basic manipulation of distributed
data, including multiple modes of functional iteration (e.g., apply()
operations). We aim to preserve consistency with base R data
structures and functions, so as to provide a simple path for users to
port computations to distributed systems.

The ddR constructs permit a user to express a wide variety of
applications, including machine-learning algorithms, that will run on
different backends. We have successfully implemented distributed
versions of algorithms such as K-means, Regression, Random Forest, and
PageRank using the ddR API. Some of these ddR algorithms are now
available on CRAN. In addition, the package provides several generic
definitions of common operators (such as colSums) that can be invoked
on distributed objects residing in the supporting backends.

Each custom ddR backend is encapsulated in its own driver package. In
the conventional style of functional OOP, the driver registers methods
for generics declared by the backend API, such that ddR can dispatch
the backend-specific instructions by only calling the generics.

The working group explored potential new backends with the aim of
broadening the applicability of the ddR interface. We hosted
presentations from external speakers on Spark and TensorFlow, and also
considered a generic SQL backend. The discussion focused on Spark
integration, and the R Consortium-funded intern Clark Fitzgerald took
on the task of developing a prototype Spark backend. The development
of the Spark backend encountered some obstacles, including the
immaturity of Spark and its R interfaces. Development is currently
paused, as we await additional funding.

During the monthly meetings, the working group deliberated on
different design improvements for ddR itself. We list two key topics
that were discussed. First, Michael Kane and Bryan Lewis argued for a
lower level API that directly operates on chunks of data. While ddR
supports chunk-wise data processing, via a combination of dmapply()
and parts(), its focus on distributed data structures means that
the chunk-based processing is exposed as the manipulation of these
data structures. Second, Clark Fitzgerald proposed restructuring the
ddR code into two layers that includes chunk-wise processing while
retaining the emphasis on distributed data structures[2]. The lower
level API, which will interface with backends, will use a Map() like
primitive to evaluate functions on chunks of data, while the higher
level ddR API will expose distributed data structures, dmapply, and
other convenience functions. This refactoring would facilitate the
implementation of additional backends.

== Discussion and Future Plans ==

The R Consortium-funded working group and internship has helped us
start a conversation on distributed computing APIs for R. The ddR
CRAN package is a concrete outcome of this working group, and serves
as a platform for exploring APIs and their integration with different
backends. While ddR is still maturing, we have arrived at a consensus
for how we should improve and finalize the ddR API.

As part of our goal for a reference implementation, we aim to develop
one or more prototype backends that will make the ddR interface useful
in practice. A good candidate backend is any open-source system that
is effective at R use cases and has strong community support. Spark
remains a viable candidate, and we also aim to further explore
TensorFlow.

We plan for a second intern to perform three tasks: (1) refactor the
ddR API to a more final form, (2) compare Spark and TensorFlow in
detail, with an eye towards the feasibility of implementing a useful
backend, and (3) implement a prototype backend based on Spark or
TensorFlow, depending on the recommendation of the working group.

By the conclusion of the working group, it will have produced:
* A stable version of the ddR package and at least one practical backend, released on CRAN,
* A list of requirements that are relevant and of interest to the community but have not yet been met by ddR, including alternative implementations that remain independent,
* A list of topics that the group believes worthy of further investigation.

[1] http://h30507.www3.hp.com/t5/Behind-the-scenes-Labs/Enhancing-R-for-Distributed-Computing/ba-p/6795535#.VjE1K7erQQj

[2] Clark Fitzgerald. https://github.com/vertica/ddR/wiki/Design

Distributed Computing Working Group Progress Report 2016

2017-05-01T18:43:10Z

MichaelLawrence: Created page with "== Introduction == Data sizes continue to increase, while single core performance has stagnated. We scale computations by leveraging multiple cores and machines. Large datas..."

== Introduction ==

Data sizes continue to increase, while single core performance has
stagnated. We scale computations by leveraging multiple cores and
machines. Large datasets are expensive to replicate, so we minimize
data movement by moving the computation to the data. Many systems,
such as Hadoop, Spark, and massively parallel processing (MPP)
databases, have emerged to support these strategies, and each exposes
its own unique interface, with little standardization.

Developing and executing an algorithm in the distributed context is a
complex task that requires specific knowledge of and dependency on the
system storing the data. It is also a task orthogonal to the primary
role of a data scientist or statistician: extracting knowledge from
data. The task thus falls to the data analysis environment, which
should mask the complexity behind a familiar interface, maintaining
user productivity. However, it is not always feasible to automatically
determine the optimal strategy for a given problem, so user input is
often beneficial. The environment should only abstract the details to
the extent deemed appropriate by the user.

R needs a standardized, layered and idiomatic abstraction for
computing on distributed data structures. R has many packages that
provide parallelism constructs as well as bridges to distributed
systems such as Hadoop. Unfortunately, each interface has its own
syntax, parallelism techniques, and supported platform(s). As a
consequence, contributors are forced to learn multiple idiosyncratic
interfaces, and to restrict each implementation to a particular
interface, thus limiting the applicability and adoption of their
software and hampering interoperability.

The idea of a unified interface stemmed from a cross-industry workshop
organized at HP Labs in early 2015. The workshop was attended by
different companies, universities, and R-core members. Immediately
after the workshop, Indrajit Roy, Edward Ma, and Michael Lawrence began
designing an abstraction that later became known as the CRAN package
ddR (Distributed Data in R)[1]. It declares a unified API for distributed
computing in R and ensures that R programs written using the API are
portable across different systems, such as Distributed R, Spark, etc.

The ddR package has completed its initial phase of development; the
first release is now on CRAN. Three ddR machine-learning algorithms
are also on CRAN, randomForest.ddR, glm.ddR, and kmeans.ddR. Two
reference backends for ddR have been completed, one for R’s parallel
package, and one for HP Distributed R. Example code and scripts to run
algorithms and code on both of these backends are available in our
public repository at https://github.com/vertica/ddR.

The overarching goal of the ddR project was for it to be a starting
point in a collaborative effort, ultimately leading to a standard API
for working with distributed data in R. We decided that it was
natural for the R Consortium to sponsor the collaboration, as it
should involve both industry and R-core members. To this end, we
established the R Consortium Working Group on Distributed Computing,
with a planned duration of a single year and the following aims:

# Agree on the goal of the group, i.e., we should have a unifying framework for distributed computing. Define success metric.
# Brainstorm on what primitives should be included in the API. We can use ddR’s API of distributed data-structures and dmapply as the starting proposal. Understand relationship with existing packages such as parallel, foreach, etc.
# Explore how ddR like interface will interact with databases. Are there connections or redundancies with dplyr and multiplyr?
# Decide on a reference implementation for the API.
# Decide on whether we should also implement a few ecosystem packages, e.g., distributed algorithms written using the API.

We declared the following milestones:

# Mid-year milestone: Finalize API. Decide who all will help with developing the top-level implementation and backends.
# End-year milestone: Summary report and reference implementation. Socialize the final package.

This report outlines the progress we have made on the above goals and
milestones, and how we plan to continue progress in the second half of
the working group term.

== Results and Current Status ==

The working group has achieved the first goal by agreeing that we
should aim for a unifying distributed computing abstraction, and we
have treated ddR as an informal API proposal.

We have discussed many of the issues related to the second goal,
deciding which primitives should be part of the API. We aim for the
API to support three shapes of data --- lists, arrays and data frames
--- and to enable the loading and basic manipulation of distributed
data, including multiple modes of functional iteration (e.g., apply()
operations). We aim to preserve consistency with base R data
structures and functions, so as to provide a simple path for users to
port computations to distributed systems.

The ddR constructs permit a user to express a wide variety of
applications, including machine-learning algorithms, that will run on
different backends. We have successfully implemented distributed
versions of algorithms such as K-means, Regression, Random Forest, and
PageRank using the ddR API. Some of these ddR algorithms are now
available on CRAN. In addition, the package provides several generic
definitions of common operators (such as colSums) that can be invoked
on distributed objects residing in the supporting backends.

Each custom ddR backend is encapsulated in its own driver package. In
the conventional style of functional OOP, the driver registers methods
for generics declared by the backend API, such that ddR can dispatch
the backend-specific instructions by only calling the generics.

The working group explored potential new backends with the aim of
broadening the applicability of the ddR interface. We hosted
presentations from external speakers on Spark and TensorFlow, and also
considered a generic SQL backend. The discussion focused on Spark
integration, and the R Consortium-funded intern Clark Fitzgerald took
on the task of developing a prototype Spark backend. The development
of the Spark backend encountered some obstacles, including the
immaturity of Spark and its R interfaces. Development is currently
paused, as we await additional funding.

During the monthly meetings, the working group deliberated on
different design improvements for ddR itself. We list two key topics
that were discussed. First, Michael Kane and Bryan Lewis argued for a
lower level API that directly operates on chunks of data. While ddR
supports chunk-wise data processing, via a combination of dmapply()
and parts(), its focus on distributed data structures means that
the chunk-based processing is exposed as the manipulation of these
data structures. Second, Clark Fitzgerald proposed restructuring the
ddR code into two layers that includes chunk-wise processing while
retaining the emphasis on distributed data structures[2]. The lower
level API, which will interface with backends, will use a Map() like
primitive to evaluate functions on chunks of data, while the higher
level ddR API will expose distributed data structures, dmapply, and
other convenience functions. This refactoring would facilitate the
implementation of additional backends.

== Discussion and Future Plans ==

The R Consortium-funded working group and internship has helped us
start a conversation on distributed computing APIs for R. The ddR
CRAN package is a concrete outcome of this working group, and serves
as a platform for exploring APIs and their integration with different
backends. While ddR is still maturing, we have arrived at a consensus
for how we should improve and finalize the ddR API.

As part of our goal for a reference implementation, we aim to develop
one or more prototype backends that will make the ddR interface useful
in practice. A good candidate backend is any open-source system that
is effective at R use cases and has strong community support. Spark
remains a viable candidate, and we also aim to further explore
TensorFlow.

We plan for a second intern to perform three tasks: (1) refactor the
ddR API to a more final form, (2) compare Spark and TensorFlow in
detail, with an eye towards the feasibility of implementing a useful
backend, and (3) implement a prototype backend based on Spark or
TensorFlow, depending on the recommendation of the working group.

By the conclusion of the working group, it will have produced:
* A stable version of the ddR package and at least one practical backend, released on CRAN,
* A list of requirements that are relevant and of interest to the community but have not yet been met by ddR, including alternative implementations that remain independent,
* A list of topics that the group believes worthy of further investigation.

[1] http://h30507.www3.hp.com/t5/Behind-the-scenes-Labs/Enhancing-R-for-Distributed-Computing/ba-p/6795535#.VjE1K7erQQj

[2] Clark Fitzgerald. https://github.com/vertica/ddR/wiki/Design

Distributed Computing Working Group

2017-05-01T18:42:48Z

MichaelLawrence:

== Goals and Purpose ==

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

== Members ==

* '''Michael Lawrence''' (Genentech)
* '''Indrajit Roy''' (HP Enterprise)
* ''Joe Rickert'' (ISC liason, RStudio)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)
* Bryan Lewis (Paradigm4)
* Hossein Falaki (databricks)

== Reports ==

[[Distributed Computing Working Group Progress Report 2016|2016 Progress Report]]

== Milestones ==

=== Achieved ===

* Adopt ddR as a prototype for a standard API for distributed computing in R

=== 2016 Internship ===

Clark Fitzgerald, a PhD student in the UC Davis Statistics department, worked on ddR and Spark integration.

* Wrote [https://github.com/clarkfitzg/sparklite sparklite] and [https://github.com/clarkfitzg/rddlist rddlist] as minimal proof-of-concept R packages to connect and store general data on Spark. [https://docs.google.com/presentation/d/1WfUQ2ockNku90GWMXonEhUEcVOWcgBmWwt5uYSSBYPY/edit?usp=sharing slides]
* [https://issues.apache.org/jira/browse/SPARK-16785 Patched SparkR] to allow user defined functions returning binary columns. This allows implementation of different data structures in SparkR.
* Updated [https://github.com/vertica/ddR/wiki/Design design documents] with suggested changes to DDR's internal design and object oriented model.
* Improved [https://github.com/vertica/ddR/pull/15 testing and ddR internals].

=== Outstanding ===

* Agree on a final standard API for distributed computing in R
* Implement at least one scalable backend based on an open-source technology like Spark, SQL, etc

== Open Questions ==

* How can we address the needs of both the end user data scientists and the algorithm implementers?
* How should we share data between R and a system like Spark?
* Is there any way to unify SparkR and sparklyr?
* Could we use the abstractions of tensorflow to partially or fully integrate with platforms like Spark?

== Minutes ==
=== 03/09/2017 ===
* Talk by Brian Lewis
** Has created a Github page with notes on Singularity.
** Singularity is a container technology for HPC applications
** No daemon. Minimum virtualization to get application running. Light weight and has very low overheads.
** Used widely in supercomputers
** All distributed computing platforms even with R skins are difficult to use.
** Containers make it much easier to abstract away the long tail of software dependencies and focus on R
** Demonstrated an example of using Singularity with Tensorflow
** Tried MPI and dopar on the 1000Genome data
** The program parses the variant data and stores chunks as files. Then ran principal components on each file.
** Overload matrix operations to use foreach/MPI underneath.
** Overall: Use existing R operators and overloading them with the appropriate backend.
** Will spend time working on Tensorflow, e.g., take a number of algorithms such as PCA and write them on top of Tensorflow using existing R primitives.

=== 12/08/2016 ===
* Yuan Tang from Uptake was the presenter
** Michael and Indrajit will write a status report for the working group sometime in December or January
** Yuan gave an overview of TensorFlow
** JJ, Dirk and Yuan are working on R layer for TensorFlow
** TensorFlow is a platform for machine learning as well as other computations (even math proofs).
** It is GPU optimized and distributed.
** It is used in search, speech recognition, Google photos, etc.
** TensorFlow computations are directed graphs. odes are operations and edges are tensors.
** A lot of array, matrix, etc. operations are available
** Backend is mostly C++. Python front end exists.
** TensorFlow R is based on the python fronted
** In multi-device setting, TensorFlow figures out which devices to use and manages communication between devices.
** Computations are fault tolerant
** Yuan has previously worked on Scikit Flow which is now TF.Learn. It’s a easy transition for Scikit learn users.
** Yuan gave a brief overview of the python interface
** TensorFlow in R handles conversion between R and Python. Syntax is very similar to python API
** Future work: Adding more examples and tutorials, integration with Kubernetes/Marathon like framework.
** During the Q/A there were questions related to whether R kernels can be supported in TensorFlow, and whether R dataframes are a natural wrapper for TensorFlow objects.

=== 11/10/2016 ===

* SparkR slides were presented by Hossein Faliki and Shivaram from Databricks and UC Berkeley:
** SparkR was a prototype from AMPLab (2014). Initially it had the RDD API and was similar to PySpark API
** In 2015, the merge with upstream Spark, the decision was made to integrate with the Dataframe API, and hide the RDD API
** In 2016 more MLLib algorithms have been integrated and new APIs have been added. A CRAN package will be released soon
** Original SparkR architecture runs R on the master that communicates with the JVM processes in the driver. the driver sends commands to the worker JVM processes, and executes them as scala/java statements.
** The system can read distributed data inside the JVM from different sources such as S3, HDFS, etc.
** The driver has a socket based connection between SparkR and the RBackend. RBackend runs on the JVM, deserializes the R code, and converts the R statements into Java calls.
** collect() and createDataFrame() are used to move data between R and JVM processes. createDataFrame will convert your local R data into a JVM based distributed data frame.
** The API has IO, Caching, MLLib, and SQL related commands
** Since Spark 2.0, we can run R processes inside the JVM worker processes. There is no need to keep long running R processes.
** There are 3 UDF functions (1) lapply, runs function on different value of a list (2) dapply, runs function on each partition of a data frame. You have to careful about how data is partitioned, and (3) gapply, performs a grouping on different column names and then runs the function on each group.
** The new CRAN package install.spark() will automatically download and install Spark. Automated CRAN checks have been added to every commit to the code. Should be available with Spark 2.1.0

* Q/A
** Currently trying to get zero copy dataframe between python and Spark. Spark 2.0 has an off heap manager that uses Arrow. Once this feature is tested on the Python API, the next step will be integration R.
** Spark dataframes gain from plan optimizations. It is not SparkR specific. R UDFs are still treated as black boxes by the optimizer
** Spark doesn't directly support matrixes. There is no immediate intent to do so either. One can store an array or vector as a single column of a Spark dataframe.

=== 10/13/2016 ===

''Detailed minutes were not taken for this meeting''

* Mario Inchiosa: Microsoft's perspective on distributed computing with R
** Microsoft R Server: abstractions and algorithms for distributed computation on top of open-source R
** Desired features of a distributed API like ddR:
*** Supports PEMA (initialize, processData, updateResults, processResults)
*** Cross-platform
*** Fast runtime
*** Supports algorithm writer and data scientist
*** Comes with a comprehensive set of algorithms
*** Easy deployment
** ddR is making good progress but does not yet meet those requirements
* Indrajit: ddR progress report and next steps
** Recap of Clark's internship
** Next step: implement some of Clark's design suggestions: https://github.com/vertica/ddR/wiki/Design
** Spark integration will be based on sparklyr
** Should we limit Spark interaction to the DataFrame API or directly interact with RDDs?
*** Consensus: will likely need flexibility of RDDs to implement everything we need, e.g., arrays and lists
** Clark and Javier raised concerns about the scalability of sharing data between R and Spark
*** Michael: Spark is a platform in its own right, so interoperability is important, should figure something out
*** Bryan Lewis: Why not use tensor abstraction from tensorflow? Spark supports tensorflow and an R interface is already in the works.
** Michael raised the issue of additional funding from the R Consortium to continue Clark's work
*** Joe Rickert suggested that the working group develop one or more white papers summarizing the findings of the working group for presentation to the Infrastructure Steering Committee.
*** Consensus was in favor of this, and several pointed out that the progress so far has been worthwhile, despite not meeting the specific goals laid out in the proposal.
* Michael: do we want to invite some external speakers, one per meeting, from groups like databricks, tensorflow, etc?
** Consensus was in favor.

=== 9/8/2016 ===

''Detailed minutes were not taken for this meeting''

* Clark Fitzgerald: internship report
** Developed two packages for low-level Spark integration: rddlist, sparklite
** Patched a bug in Spark
** ddR needs refactoring before Spark integration is feasible:
*** dlist, dframe, and darray should be formal classes.
*** Partitions of data should be represented by a distributed list abstraction, and most functions (e.g., dmapply) should be implemented on top of that list.
* Javier: sparklyr update
** Preparing for CRAN release
** Mario: what happened to sparkapi?
*** Javier: sparkapi has been merged into sparklyr in order to avoid overhead of maintaining two packages. ddR can do everything it needs with sparklyr.
* Luke Tierney: Update on the low-level vector abstraction, which might support interfaces like ddR and sparklyr.
** Overall approach seems feasible, but still working out a few details.
** Will land in a branch soon.
* Bernd Bischl: update on the batchtools package
** Successor to BatchJobs based on in-memory database

=== 8/11/2016 ===

''Meeting was canceled due to lack of availability.''

=== 7/14/2016 ===

* Introduced Clark who is the intern funded by R Consortium. Clark is a graduate student from UC Davis. He will work on ddR integration with Spark and improving the core ddR API as well such as adding a distributed apply() for matrices, split function, etc.
* Bernd: Can I play around with ddR now? What backend should I use? How robust is the code?
** Clark: It's in good enough shape to be played around with. We will continue to improve it. Hopefully the spark integration will be done before the end of my internship in September.
* Q: Is anyone working on using ddR to make ML scale better.
** Indrajit: We have kmeans, glm, etc. already in CRAN.
** Michael Kane: We are working on glmnet and other packages related to algorithm development.
* Javier gave a demo of sparklyr and sparkapi.
** Motivation for the pacakage: The SparkR package overrides the dplyr interface. This is an issue for RStudio. SparkR is not a CRAN package which makes it difficult to add changes. dplyr is the most popular tool by RStudio and is broken on SparkR.
** Sparklyr provides a dplyr interface. It will also support ML like interfaces, such as consuming a ML model.
** Sparklyr does not currently support any distributed computing features. Instead we can recommend ddR as the distributed computing framework on top of sparkapi. We will put the code in CRAN in a couple of weeks.
** Simon: Can you talk more about the wrapper/low level API to work with Spark?
*** Javier: The package underneath the cover is called "sparkapi" it is to be used by pacakge builders. "spark_context()" and "invoke()" are the functionality to call scala methods. It does not you to currently run R user defined functions. I am currently working on enabling that feature. Depending upon the interest in using ddR with sparkapi, I can spend more time to make sparkapi feature rich.
** Indrajit: What versions of Spark are supported
*** Javier: Anything after 1.6
** Bernd: How do you export data?
*** Javier: We are using all the code from SparkR. So everything in SparkR should continue to work. We don't need to change SparkR. We just need to maintain compatibility.
** Bernd: What happens when the RDDs are very large?
*** Javier: Spark will spill on disk.
* Michael Kane: Presented examples that he implemented on ddR.
** Talked about how the different distributed packages compare to each other in terms of functionality.
** Michael K. looked at glm and truncated SVD on ddR. Was able to implement irls on ddR by implementing two distributed functions such as "cross". In truncated SVD only needed to overload two different distributed multiplications.
** Ran these algorithms on the 1000 genome dataset.
** Overall liked ddR since it was easy to implement the algorithms in the package.
** New ideas:
*** Trying to separate the data layer from the execution layer
*** Create an API that works on "chunks" (which is similar to the "parts" API in ddR). Would like to add these APIs to ddR.
*** Indrajit: You should be able to get some of the chunk like features by using parts and dmapply. E.g., you can call dmapply to read 10 different files, which correspond 10 chunks now. These are however wrapped as a darray or dframe. But you can continue to work on the individual chunks by using parts(i).

=== 6/2/2016 ===

* Round table introduction
* (Michael) Goals for the group:
** Make a common abstraction/interfaces to make it easier to work with distributed data and R
** Unify the interface
** Working group will run for a year. Get an API defined, get at least one open source reference implementations
** not everyone needs to work hands on. We will create smaller groups to focus on those aspects.
** We tried to get a diverse group of participants
* Logistics: meet monthly, focus groups may meet more often
* R Consoritum may be able to figure ways to fund smaller projects that come out of the working group
* Michael Kane: Should we start with an inventory of what is available and people are using?
** Michael Lawrence: Yes, we should find the collection of tools as well as the use cases that are common.
** Joe: I will figure out a wiki space.
* Javier: Who are the end users? Simon: Common layer needed to get algorithms working. We started from algos and tried to find the minimal common api. One of the goals is to make sure everyone is on the same page and not trying to create his/her own custom interface.
* Javier: Should we try to get people with more algo expertise?
* Joe: Simon do you have a stack diagram?
* Simon: Can we get R Consortium to help write things up and draw things?
* Next meeting: Javier is going to present SparkR next time.

Ddr2016

2017-05-01T17:59:37Z

MichaelLawrence: Created page with "= Distributed Computing Working Group Progress Report: 2016 = == Introduction == Data sizes continue to increase, while single core performance has stagnated. We scale comp..."

= Distributed Computing Working Group Progress Report: 2016 =

== Introduction ==

Data sizes continue to increase, while single core performance has
stagnated. We scale computations by leveraging multiple cores and
machines. Large datasets are expensive to replicate, so we minimize
data movement by moving the computation to the data. Many systems,
such as Hadoop, Spark, and massively parallel processing (MPP)
databases, have emerged to support these strategies, and each exposes
its own unique interface, with little standardization.

Developing and executing an algorithm in the distributed context is a
complex task that requires specific knowledge of and dependency on the
system storing the data. It is also a task orthogonal to the primary
role of a data scientist or statistician: extracting knowledge from
data. The task thus falls to the data analysis environment, which
should mask the complexity behind a familiar interface, maintaining
user productivity. However, it is not always feasible to automatically
determine the optimal strategy for a given problem, so user input is
often beneficial. The environment should only abstract the details to
the extent deemed appropriate by the user.

R needs a standardized, layered and idiomatic abstraction for
computing on distributed data structures. R has many packages that
provide parallelism constructs as well as bridges to distributed
systems such as Hadoop. Unfortunately, each interface has its own
syntax, parallelism techniques, and supported platform(s). As a
consequence, contributors are forced to learn multiple idiosyncratic
interfaces, and to restrict each implementation to a particular
interface, thus limiting the applicability and adoption of their
software and hampering interoperability.

The idea of a unified interface stemmed from a cross-industry workshop
organized at HP Labs in early 2015. The workshop was attended by
different companies, universities, and R-core members. Immediately
after the workshop, Indrajit Roy, Edward Ma, and Michael Lawrence began
designing an abstraction that later became known as the CRAN package
ddR (Distributed Data in R)[1]. It declares a unified API for distributed
computing in R and ensures that R programs written using the API are
portable across different systems, such as Distributed R, Spark, etc.

The ddR package has completed its initial phase of development; the
first release is now on CRAN. Three ddR machine-learning algorithms
are also on CRAN, randomForest.ddR, glm.ddR, and kmeans.ddR. Two
reference backends for ddR have been completed, one for R’s parallel
package, and one for HP Distributed R. Example code and scripts to run
algorithms and code on both of these backends are available in our
public repository at https://github.com/vertica/ddR.

The overarching goal of the ddR project was for it to be a starting
point in a collaborative effort, ultimately leading to a standard API
for working with distributed data in R. We decided that it was
natural for the R Consortium to sponsor the collaboration, as it
should involve both industry and R-core members. To this end, we
established the R Consortium Working Group on Distributed Computing,
with a planned duration of a single year and the following aims:

# Agree on the goal of the group, i.e., we should have a unifying framework for distributed computing. Define success metric.
# Brainstorm on what primitives should be included in the API. We can use ddR’s API of distributed data-structures and dmapply as the starting proposal. Understand relationship with existing packages such as parallel, foreach, etc.
# Explore how ddR like interface will interact with databases. Are there connections or redundancies with dplyr and multiplyr?
# Decide on a reference implementation for the API.
# Decide on whether we should also implement a few ecosystem packages, e.g., distributed algorithms written using the API.

We declared the following milestones:

# Mid-year milestone: Finalize API. Decide who all will help with developing the top-level implementation and backends.
# End-year milestone: Summary report and reference implementation. Socialize the final package.

This report outlines the progress we have made on the above goals and
milestones, and how we plan to continue progress in the second half of
the working group term.

== Results and Current Status ==

The working group has achieved the first goal by agreeing that we
should aim for a unifying distributed computing abstraction, and we
have treated ddR as an informal API proposal.

We have discussed many of the issues related to the second goal,
deciding which primitives should be part of the API. We aim for the
API to support three shapes of data --- lists, arrays and data frames
--- and to enable the loading and basic manipulation of distributed
data, including multiple modes of functional iteration (e.g., apply()
operations). We aim to preserve consistency with base R data
structures and functions, so as to provide a simple path for users to
port computations to distributed systems.

The ddR constructs permit a user to express a wide variety of
applications, including machine-learning algorithms, that will run on
different backends. We have successfully implemented distributed
versions of algorithms such as K-means, Regression, Random Forest, and
PageRank using the ddR API. Some of these ddR algorithms are now
available on CRAN. In addition, the package provides several generic
definitions of common operators (such as colSums) that can be invoked
on distributed objects residing in the supporting backends.

Each custom ddR backend is encapsulated in its own driver package. In
the conventional style of functional OOP, the driver registers methods
for generics declared by the backend API, such that ddR can dispatch
the backend-specific instructions by only calling the generics.

The working group explored potential new backends with the aim of
broadening the applicability of the ddR interface. We hosted
presentations from external speakers on Spark and TensorFlow, and also
considered a generic SQL backend. The discussion focused on Spark
integration, and the R Consortium-funded intern Clark Fitzgerald took
on the task of developing a prototype Spark backend. The development
of the Spark backend encountered some obstacles, including the
immaturity of Spark and its R interfaces. Development is currently
paused, as we await additional funding.

During the monthly meetings, the working group deliberated on
different design improvements for ddR itself. We list two key topics
that were discussed. First, Michael Kane and Bryan Lewis argued for a
lower level API that directly operates on chunks of data. While ddR
supports chunk-wise data processing, via a combination of dmapply()
and parts(), its focus on distributed data structures means that
the chunk-based processing is exposed as the manipulation of these
data structures. Second, Clark Fitzgerald proposed restructuring the
ddR code into two layers that includes chunk-wise processing while
retaining the emphasis on distributed data structures[2]. The lower
level API, which will interface with backends, will use a Map() like
primitive to evaluate functions on chunks of data, while the higher
level ddR API will expose distributed data structures, dmapply, and
other convenience functions. This refactoring would facilitate the
implementation of additional backends.

== Discussion and Future Plans ==

The R Consortium-funded working group and internship has helped us
start a conversation on distributed computing APIs for R. The ddR
CRAN package is a concrete outcome of this working group, and serves
as a platform for exploring APIs and their integration with different
backends. While ddR is still maturing, we have arrived at a consensus
for how we should improve and finalize the ddR API.

As part of our goal for a reference implementation, we aim to develop
one or more prototype backends that will make the ddR interface useful
in practice. A good candidate backend is any open-source system that
is effective at R use cases and has strong community support. Spark
remains a viable candidate, and we also aim to further explore
TensorFlow.

We plan for a second intern to perform three tasks: (1) refactor the
ddR API to a more final form, (2) compare Spark and TensorFlow in
detail, with an eye towards the feasibility of implementing a useful
backend, and (3) implement a prototype backend based on Spark or
TensorFlow, depending on the recommendation of the working group.

By the conclusion of the working group, it will have produced:
* A stable version of the ddR package and at least one practical backend, released on CRAN,
* A list of requirements that are relevant and of interest to the community but have not yet been met by ddR, including alternative implementations that remain independent,
* A list of topics that the group believes worthy of further investigation.

[1] http://h30507.www3.hp.com/t5/Behind-the-scenes-Labs/Enhancing-R-for-Distributed-Computing/ba-p/6795535#.VjE1K7erQQj

[2] Clark Fitzgerald. https://github.com/vertica/ddR/wiki/Design

Distributed Computing Working Group

2017-05-01T17:56:12Z

MichaelLawrence:

== Goals and Purpose ==

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

== Members ==

* '''Michael Lawrence''' (Genentech)
* '''Indrajit Roy''' (HP Enterprise)
* ''Joe Rickert'' (ISC liason, RStudio)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)
* Bryan Lewis (Paradigm4)
* Hossein Falaki (databricks)

== Reports ==

[[ddr2016|2016 Progress Report]]

== Milestones ==

=== Achieved ===

* Adopt ddR as a prototype for a standard API for distributed computing in R

=== 2016 Internship ===

Clark Fitzgerald, a PhD student in the UC Davis Statistics department, worked on ddR and Spark integration.

* Wrote [https://github.com/clarkfitzg/sparklite sparklite] and [https://github.com/clarkfitzg/rddlist rddlist] as minimal proof-of-concept R packages to connect and store general data on Spark. [https://docs.google.com/presentation/d/1WfUQ2ockNku90GWMXonEhUEcVOWcgBmWwt5uYSSBYPY/edit?usp=sharing slides]
* [https://issues.apache.org/jira/browse/SPARK-16785 Patched SparkR] to allow user defined functions returning binary columns. This allows implementation of different data structures in SparkR.
* Updated [https://github.com/vertica/ddR/wiki/Design design documents] with suggested changes to DDR's internal design and object oriented model.
* Improved [https://github.com/vertica/ddR/pull/15 testing and ddR internals].

=== Outstanding ===

* Agree on a final standard API for distributed computing in R
* Implement at least one scalable backend based on an open-source technology like Spark, SQL, etc

== Open Questions ==

* How can we address the needs of both the end user data scientists and the algorithm implementers?
* How should we share data between R and a system like Spark?
* Is there any way to unify SparkR and sparklyr?
* Could we use the abstractions of tensorflow to partially or fully integrate with platforms like Spark?

== Minutes ==
=== 03/09/2017 ===
* Talk by Brian Lewis
** Has created a Github page with notes on Singularity.
** Singularity is a container technology for HPC applications
** No daemon. Minimum virtualization to get application running. Light weight and has very low overheads.
** Used widely in supercomputers
** All distributed computing platforms even with R skins are difficult to use.
** Containers make it much easier to abstract away the long tail of software dependencies and focus on R
** Demonstrated an example of using Singularity with Tensorflow
** Tried MPI and dopar on the 1000Genome data
** The program parses the variant data and stores chunks as files. Then ran principal components on each file.
** Overload matrix operations to use foreach/MPI underneath.
** Overall: Use existing R operators and overloading them with the appropriate backend.
** Will spend time working on Tensorflow, e.g., take a number of algorithms such as PCA and write them on top of Tensorflow using existing R primitives.

=== 12/08/2016 ===
* Yuan Tang from Uptake was the presenter
** Michael and Indrajit will write a status report for the working group sometime in December or January
** Yuan gave an overview of TensorFlow
** JJ, Dirk and Yuan are working on R layer for TensorFlow
** TensorFlow is a platform for machine learning as well as other computations (even math proofs).
** It is GPU optimized and distributed.
** It is used in search, speech recognition, Google photos, etc.
** TensorFlow computations are directed graphs. odes are operations and edges are tensors.
** A lot of array, matrix, etc. operations are available
** Backend is mostly C++. Python front end exists.
** TensorFlow R is based on the python fronted
** In multi-device setting, TensorFlow figures out which devices to use and manages communication between devices.
** Computations are fault tolerant
** Yuan has previously worked on Scikit Flow which is now TF.Learn. It’s a easy transition for Scikit learn users.
** Yuan gave a brief overview of the python interface
** TensorFlow in R handles conversion between R and Python. Syntax is very similar to python API
** Future work: Adding more examples and tutorials, integration with Kubernetes/Marathon like framework.
** During the Q/A there were questions related to whether R kernels can be supported in TensorFlow, and whether R dataframes are a natural wrapper for TensorFlow objects.

=== 11/10/2016 ===

* SparkR slides were presented by Hossein Faliki and Shivaram from Databricks and UC Berkeley:
** SparkR was a prototype from AMPLab (2014). Initially it had the RDD API and was similar to PySpark API
** In 2015, the merge with upstream Spark, the decision was made to integrate with the Dataframe API, and hide the RDD API
** In 2016 more MLLib algorithms have been integrated and new APIs have been added. A CRAN package will be released soon
** Original SparkR architecture runs R on the master that communicates with the JVM processes in the driver. the driver sends commands to the worker JVM processes, and executes them as scala/java statements.
** The system can read distributed data inside the JVM from different sources such as S3, HDFS, etc.
** The driver has a socket based connection between SparkR and the RBackend. RBackend runs on the JVM, deserializes the R code, and converts the R statements into Java calls.
** collect() and createDataFrame() are used to move data between R and JVM processes. createDataFrame will convert your local R data into a JVM based distributed data frame.
** The API has IO, Caching, MLLib, and SQL related commands
** Since Spark 2.0, we can run R processes inside the JVM worker processes. There is no need to keep long running R processes.
** There are 3 UDF functions (1) lapply, runs function on different value of a list (2) dapply, runs function on each partition of a data frame. You have to careful about how data is partitioned, and (3) gapply, performs a grouping on different column names and then runs the function on each group.
** The new CRAN package install.spark() will automatically download and install Spark. Automated CRAN checks have been added to every commit to the code. Should be available with Spark 2.1.0

* Q/A
** Currently trying to get zero copy dataframe between python and Spark. Spark 2.0 has an off heap manager that uses Arrow. Once this feature is tested on the Python API, the next step will be integration R.
** Spark dataframes gain from plan optimizations. It is not SparkR specific. R UDFs are still treated as black boxes by the optimizer
** Spark doesn't directly support matrixes. There is no immediate intent to do so either. One can store an array or vector as a single column of a Spark dataframe.

=== 10/13/2016 ===

''Detailed minutes were not taken for this meeting''

* Mario Inchiosa: Microsoft's perspective on distributed computing with R
** Microsoft R Server: abstractions and algorithms for distributed computation on top of open-source R
** Desired features of a distributed API like ddR:
*** Supports PEMA (initialize, processData, updateResults, processResults)
*** Cross-platform
*** Fast runtime
*** Supports algorithm writer and data scientist
*** Comes with a comprehensive set of algorithms
*** Easy deployment
** ddR is making good progress but does not yet meet those requirements
* Indrajit: ddR progress report and next steps
** Recap of Clark's internship
** Next step: implement some of Clark's design suggestions: https://github.com/vertica/ddR/wiki/Design
** Spark integration will be based on sparklyr
** Should we limit Spark interaction to the DataFrame API or directly interact with RDDs?
*** Consensus: will likely need flexibility of RDDs to implement everything we need, e.g., arrays and lists
** Clark and Javier raised concerns about the scalability of sharing data between R and Spark
*** Michael: Spark is a platform in its own right, so interoperability is important, should figure something out
*** Bryan Lewis: Why not use tensor abstraction from tensorflow? Spark supports tensorflow and an R interface is already in the works.
** Michael raised the issue of additional funding from the R Consortium to continue Clark's work
*** Joe Rickert suggested that the working group develop one or more white papers summarizing the findings of the working group for presentation to the Infrastructure Steering Committee.
*** Consensus was in favor of this, and several pointed out that the progress so far has been worthwhile, despite not meeting the specific goals laid out in the proposal.
* Michael: do we want to invite some external speakers, one per meeting, from groups like databricks, tensorflow, etc?
** Consensus was in favor.

=== 9/8/2016 ===

''Detailed minutes were not taken for this meeting''

* Clark Fitzgerald: internship report
** Developed two packages for low-level Spark integration: rddlist, sparklite
** Patched a bug in Spark
** ddR needs refactoring before Spark integration is feasible:
*** dlist, dframe, and darray should be formal classes.
*** Partitions of data should be represented by a distributed list abstraction, and most functions (e.g., dmapply) should be implemented on top of that list.
* Javier: sparklyr update
** Preparing for CRAN release
** Mario: what happened to sparkapi?
*** Javier: sparkapi has been merged into sparklyr in order to avoid overhead of maintaining two packages. ddR can do everything it needs with sparklyr.
* Luke Tierney: Update on the low-level vector abstraction, which might support interfaces like ddR and sparklyr.
** Overall approach seems feasible, but still working out a few details.
** Will land in a branch soon.
* Bernd Bischl: update on the batchtools package
** Successor to BatchJobs based on in-memory database

=== 8/11/2016 ===

''Meeting was canceled due to lack of availability.''

=== 7/14/2016 ===

* Introduced Clark who is the intern funded by R Consortium. Clark is a graduate student from UC Davis. He will work on ddR integration with Spark and improving the core ddR API as well such as adding a distributed apply() for matrices, split function, etc.
* Bernd: Can I play around with ddR now? What backend should I use? How robust is the code?
** Clark: It's in good enough shape to be played around with. We will continue to improve it. Hopefully the spark integration will be done before the end of my internship in September.
* Q: Is anyone working on using ddR to make ML scale better.
** Indrajit: We have kmeans, glm, etc. already in CRAN.
** Michael Kane: We are working on glmnet and other packages related to algorithm development.
* Javier gave a demo of sparklyr and sparkapi.
** Motivation for the pacakage: The SparkR package overrides the dplyr interface. This is an issue for RStudio. SparkR is not a CRAN package which makes it difficult to add changes. dplyr is the most popular tool by RStudio and is broken on SparkR.
** Sparklyr provides a dplyr interface. It will also support ML like interfaces, such as consuming a ML model.
** Sparklyr does not currently support any distributed computing features. Instead we can recommend ddR as the distributed computing framework on top of sparkapi. We will put the code in CRAN in a couple of weeks.
** Simon: Can you talk more about the wrapper/low level API to work with Spark?
*** Javier: The package underneath the cover is called "sparkapi" it is to be used by pacakge builders. "spark_context()" and "invoke()" are the functionality to call scala methods. It does not you to currently run R user defined functions. I am currently working on enabling that feature. Depending upon the interest in using ddR with sparkapi, I can spend more time to make sparkapi feature rich.
** Indrajit: What versions of Spark are supported
*** Javier: Anything after 1.6
** Bernd: How do you export data?
*** Javier: We are using all the code from SparkR. So everything in SparkR should continue to work. We don't need to change SparkR. We just need to maintain compatibility.
** Bernd: What happens when the RDDs are very large?
*** Javier: Spark will spill on disk.
* Michael Kane: Presented examples that he implemented on ddR.
** Talked about how the different distributed packages compare to each other in terms of functionality.
** Michael K. looked at glm and truncated SVD on ddR. Was able to implement irls on ddR by implementing two distributed functions such as "cross". In truncated SVD only needed to overload two different distributed multiplications.
** Ran these algorithms on the 1000 genome dataset.
** Overall liked ddR since it was easy to implement the algorithms in the package.
** New ideas:
*** Trying to separate the data layer from the execution layer
*** Create an API that works on "chunks" (which is similar to the "parts" API in ddR). Would like to add these APIs to ddR.
*** Indrajit: You should be able to get some of the chunk like features by using parts and dmapply. E.g., you can call dmapply to read 10 different files, which correspond 10 chunks now. These are however wrapped as a darray or dframe. But you can continue to work on the individual chunks by using parts(i).

=== 6/2/2016 ===

* Round table introduction
* (Michael) Goals for the group:
** Make a common abstraction/interfaces to make it easier to work with distributed data and R
** Unify the interface
** Working group will run for a year. Get an API defined, get at least one open source reference implementations
** not everyone needs to work hands on. We will create smaller groups to focus on those aspects.
** We tried to get a diverse group of participants
* Logistics: meet monthly, focus groups may meet more often
* R Consoritum may be able to figure ways to fund smaller projects that come out of the working group
* Michael Kane: Should we start with an inventory of what is available and people are using?
** Michael Lawrence: Yes, we should find the collection of tools as well as the use cases that are common.
** Joe: I will figure out a wiki space.
* Javier: Who are the end users? Simon: Common layer needed to get algorithms working. We started from algos and tried to find the minimal common api. One of the goals is to make sure everyone is on the same page and not trying to create his/her own custom interface.
* Javier: Should we try to get people with more algo expertise?
* Joe: Simon do you have a stack diagram?
* Simon: Can we get R Consortium to help write things up and draw things?
* Next meeting: Javier is going to present SparkR next time.

Distributed Computing Working Group

2016-12-09T18:48:40Z

MichaelLawrence: Yuan works at Uptake

== Goals and Purpose ==

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

== Members ==

* '''Michael Lawrence''' (Genentech)
* '''Indrajit Roy''' (HP Enterprise)
* ''Joe Rickert'' (ISC liason, RStudio)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)
* Bryan Lewis (Paradigm4)
* Hossein Falaki (databricks)

== Milestones ==

=== Achieved ===

* Adopt ddR as a prototype for a standard API for distributed computing in R

=== 2016 Internship ===

Clark Fitzgerald, a PhD student in the UC Davis Statistics department, worked on ddR and Spark integration.

* Wrote [https://github.com/clarkfitzg/sparklite sparklite] and [https://github.com/clarkfitzg/rddlist rddlist] as minimal proof-of-concept R packages to connect and store general data on Spark. [https://docs.google.com/presentation/d/1WfUQ2ockNku90GWMXonEhUEcVOWcgBmWwt5uYSSBYPY/edit?usp=sharing slides]
* [https://issues.apache.org/jira/browse/SPARK-16785 Patched SparkR] to allow user defined functions returning binary columns. This allows implementation of different data structures in SparkR.
* Updated [https://github.com/vertica/ddR/wiki/Design design documents] with suggested changes to DDR's internal design and object oriented model.
* Improved [https://github.com/vertica/ddR/pull/15 testing and ddR internals].

=== Outstanding ===

* Agree on a final standard API for distributed computing in R
* Implement at least one scalable backend based on an open-source technology like Spark, SQL, etc

== Open Questions ==

* How can we address the needs of both the end user data scientists and the algorithm implementers?
* How should we share data between R and a system like Spark?
* Is there any way to unify SparkR and sparklyr?
* Could we use the abstractions of tensorflow to partially or fully integrate with platforms like Spark?

== Minutes ==
=== 12/08/2016 ===
* Yuan Tang from Uptake was the presenter
** Michael and Indrajit will write a status report for the working group sometime in December or January
** Yuan gave an overview of TensorFlow
** JJ, Dirk and Yuan are working on R layer for TensorFlow
** TensorFlow is a platform for machine learning as well as other computations (even math proofs).
** It is GPU optimized and distributed.
** It is used in search, speech recognition, Google photos, etc.
** TensorFlow computations are directed graphs. odes are operations and edges are tensors.
** A lot of array, matrix, etc. operations are available
** Backend is mostly C++. Python front end exists.
** TensorFlow R is based on the python fronted
** In multi-device setting, TensorFlow figures out which devices to use and manages communication between devices.
** Computations are fault tolerant
** Yuan has previously worked on Scikit Flow which is now TF.Learn. It’s a easy transition for Scikit learn users.
** Yuan gave a brief overview of the python interface
** TensorFlow in R handles conversion between R and Python. Syntax is very similar to python API
** Future work: Adding more examples and tutorials, integration with Kubernetes/Marathon like framework.
** During the Q/A there were questions related to whether R kernels can be supported in TensorFlow, and whether R dataframes are a natural wrapper for TensorFlow objects.

=== 11/10/2016 ===

* SparkR slides were presented by Hossein Faliki and Shivaram from Databricks and UC Berkeley:
** SparkR was a prototype from AMPLab (2014). Initially it had the RDD API and was similar to PySpark API
** In 2015, the merge with upstream Spark, the decision was made to integrate with the Dataframe API, and hide the RDD API
** In 2016 more MLLib algorithms have been integrated and new APIs have been added. A CRAN package will be released soon
** Original SparkR architecture runs R on the master that communicates with the JVM processes in the driver. the driver sends commands to the worker JVM processes, and executes them as scala/java statements.
** The system can read distributed data inside the JVM from different sources such as S3, HDFS, etc.
** The driver has a socket based connection between SparkR and the RBackend. RBackend runs on the JVM, deserializes the R code, and converts the R statements into Java calls.
** collect() and createDataFrame() are used to move data between R and JVM processes. createDataFrame will convert your local R data into a JVM based distributed data frame.
** The API has IO, Caching, MLLib, and SQL related commands
** Since Spark 2.0, we can run R processes inside the JVM worker processes. There is no need to keep long running R processes.
** There are 3 UDF functions (1) lapply, runs function on different value of a list (2) dapply, runs function on each partition of a data frame. You have to careful about how data is partitioned, and (3) gapply, performs a grouping on different column names and then runs the function on each group.
** The new CRAN package install.spark() will automatically download and install Spark. Automated CRAN checks have been added to every commit to the code. Should be available with Spark 2.1.0

* Q/A
** Currently trying to get zero copy dataframe between python and Spark. Spark 2.0 has an off heap manager that uses Arrow. Once this feature is tested on the Python API, the next step will be integration R.
** Spark dataframes gain from plan optimizations. It is not SparkR specific. R UDFs are still treated as black boxes by the optimizer
** Spark doesn't directly support matrixes. There is no immediate intent to do so either. One can store an array or vector as a single column of a Spark dataframe.

=== 10/13/2016 ===

''Detailed minutes were not taken for this meeting''

* Mario Inchiosa: Microsoft's perspective on distributed computing with R
** Microsoft R Server: abstractions and algorithms for distributed computation on top of open-source R
** Desired features of a distributed API like ddR:
*** Supports PEMA (initialize, processData, updateResults, processResults)
*** Cross-platform
*** Fast runtime
*** Supports algorithm writer and data scientist
*** Comes with a comprehensive set of algorithms
*** Easy deployment
** ddR is making good progress but does not yet meet those requirements
* Indrajit: ddR progress report and next steps
** Recap of Clark's internship
** Next step: implement some of Clark's design suggestions: https://github.com/vertica/ddR/wiki/Design
** Spark integration will be based on sparklyr
** Should we limit Spark interaction to the DataFrame API or directly interact with RDDs?
*** Consensus: will likely need flexibility of RDDs to implement everything we need, e.g., arrays and lists
** Clark and Javier raised concerns about the scalability of sharing data between R and Spark
*** Michael: Spark is a platform in its own right, so interoperability is important, should figure something out
*** Bryan Lewis: Why not use tensor abstraction from tensorflow? Spark supports tensorflow and an R interface is already in the works.
** Michael raised the issue of additional funding from the R Consortium to continue Clark's work
*** Joe Rickert suggested that the working group develop one or more white papers summarizing the findings of the working group for presentation to the Infrastructure Steering Committee.
*** Consensus was in favor of this, and several pointed out that the progress so far has been worthwhile, despite not meeting the specific goals laid out in the proposal.
* Michael: do we want to invite some external speakers, one per meeting, from groups like databricks, tensorflow, etc?
** Consensus was in favor.

=== 9/8/2016 ===

''Detailed minutes were not taken for this meeting''

* Clark Fitzgerald: internship report
** Developed two packages for low-level Spark integration: rddlist, sparklite
** Patched a bug in Spark
** ddR needs refactoring before Spark integration is feasible:
*** dlist, dframe, and darray should be formal classes.
*** Partitions of data should be represented by a distributed list abstraction, and most functions (e.g., dmapply) should be implemented on top of that list.
* Javier: sparklyr update
** Preparing for CRAN release
** Mario: what happened to sparkapi?
*** Javier: sparkapi has been merged into sparklyr in order to avoid overhead of maintaining two packages. ddR can do everything it needs with sparklyr.
* Luke Tierney: Update on the low-level vector abstraction, which might support interfaces like ddR and sparklyr.
** Overall approach seems feasible, but still working out a few details.
** Will land in a branch soon.
* Bernd Bischl: update on the batchtools package
** Successor to BatchJobs based on in-memory database

=== 8/11/2016 ===

''Meeting was canceled due to lack of availability.''

=== 7/14/2016 ===

* Introduced Clark who is the intern funded by R Consortium. Clark is a graduate student from UC Davis. He will work on ddR integration with Spark and improving the core ddR API as well such as adding a distributed apply() for matrices, split function, etc.
* Bernd: Can I play around with ddR now? What backend should I use? How robust is the code?
** Clark: It's in good enough shape to be played around with. We will continue to improve it. Hopefully the spark integration will be done before the end of my internship in September.
* Q: Is anyone working on using ddR to make ML scale better.
** Indrajit: We have kmeans, glm, etc. already in CRAN.
** Michael Kane: We are working on glmnet and other packages related to algorithm development.
* Javier gave a demo of sparklyr and sparkapi.
** Motivation for the pacakage: The SparkR package overrides the dplyr interface. This is an issue for RStudio. SparkR is not a CRAN package which makes it difficult to add changes. dplyr is the most popular tool by RStudio and is broken on SparkR.
** Sparklyr provides a dplyr interface. It will also support ML like interfaces, such as consuming a ML model.
** Sparklyr does not currently support any distributed computing features. Instead we can recommend ddR as the distributed computing framework on top of sparkapi. We will put the code in CRAN in a couple of weeks.
** Simon: Can you talk more about the wrapper/low level API to work with Spark?
*** Javier: The package underneath the cover is called "sparkapi" it is to be used by pacakge builders. "spark_context()" and "invoke()" are the functionality to call scala methods. It does not you to currently run R user defined functions. I am currently working on enabling that feature. Depending upon the interest in using ddR with sparkapi, I can spend more time to make sparkapi feature rich.
** Indrajit: What versions of Spark are supported
*** Javier: Anything after 1.6
** Bernd: How do you export data?
*** Javier: We are using all the code from SparkR. So everything in SparkR should continue to work. We don't need to change SparkR. We just need to maintain compatibility.
** Bernd: What happens when the RDDs are very large?
*** Javier: Spark will spill on disk.
* Michael Kane: Presented examples that he implemented on ddR.
** Talked about how the different distributed packages compare to each other in terms of functionality.
** Michael K. looked at glm and truncated SVD on ddR. Was able to implement irls on ddR by implementing two distributed functions such as "cross". In truncated SVD only needed to overload two different distributed multiplications.
** Ran these algorithms on the 1000 genome dataset.
** Overall liked ddR since it was easy to implement the algorithms in the package.
** New ideas:
*** Trying to separate the data layer from the execution layer
*** Create an API that works on "chunks" (which is similar to the "parts" API in ddR). Would like to add these APIs to ddR.
*** Indrajit: You should be able to get some of the chunk like features by using parts and dmapply. E.g., you can call dmapply to read 10 different files, which correspond 10 chunks now. These are however wrapped as a darray or dframe. But you can continue to work on the individual chunks by using parts(i).

=== 6/2/2016 ===

* Round table introduction
* (Michael) Goals for the group:
** Make a common abstraction/interfaces to make it easier to work with distributed data and R
** Unify the interface
** Working group will run for a year. Get an API defined, get at least one open source reference implementations
** not everyone needs to work hands on. We will create smaller groups to focus on those aspects.
** We tried to get a diverse group of participants
* Logistics: meet monthly, focus groups may meet more often
* R Consoritum may be able to figure ways to fund smaller projects that come out of the working group
* Michael Kane: Should we start with an inventory of what is available and people are using?
** Michael Lawrence: Yes, we should find the collection of tools as well as the use cases that are common.
** Joe: I will figure out a wiki space.
* Javier: Who are the end users? Simon: Common layer needed to get algorithms working. We started from algos and tried to find the minimal common api. One of the goals is to make sure everyone is on the same page and not trying to create his/her own custom interface.
* Javier: Should we try to get people with more algo expertise?
* Joe: Simon do you have a stack diagram?
* Simon: Can we get R Consortium to help write things up and draw things?
* Next meeting: Javier is going to present SparkR next time.

Distributed Computing Working Group

2016-11-10T21:14:05Z

MichaelLawrence: add Hossein Falaki

Distributed Computing Working Group

2016-10-18T16:52:07Z

MichaelLawrence: change Joe's affiliation

Distributed Computing Working Group

2016-10-18T16:27:16Z

MichaelLawrence:

Distributed Computing Working Group

2016-10-18T11:52:48Z

MichaelLawrence: October minutes

Distributed Computing Working Group

2016-10-18T11:29:29Z

MichaelLawrence: September minutes

Distributed Computing Working Group

2016-10-18T11:11:44Z

MichaelLawrence: add July minutes

Distributed Computing Working Group

2016-07-14T12:39:43Z

MichaelLawrence:

Distributed Computing Working Group

2016-06-24T15:44:58Z

MichaelLawrence:

Distributed Computing Working Group

2016-06-24T15:43:59Z

MichaelLawrence:

Distributed Computing Working Group

2016-06-24T15:34:25Z

MichaelLawrence:

Goals and Purpose

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

Members

* '''Michael Lawrence''' (Genentech)
* '''Indrajit Roy''' (HP Enterprise)
* ''Joe Rickert'' (Microsoft)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)

Minutes

Distributed Computing Working Group

2016-06-24T15:34:06Z

MichaelLawrence:

Goals and Purpose

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

Members

* '''Michael Lawrence''' (Genentech)
* ''''Indrajit Roy'''' (HP Enterprise)
* ''Joe Rickert'' (Microsoft)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)

Minutes

Distributed Computing Working Group

2016-06-24T15:33:49Z

MichaelLawrence:

Goals and Purpose

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

Members

* '''Michael Lawrence''' (Genentech)
* ""Indrajit Roy"" (HP Enterprise)
* ''Joe Rickert'' (Microsoft)
* Bernd Bischl (LMU)
* Matt Dowle (H2O)
* Mario Inchiosa (Microsoft)
* Michael Kane (Yale)
* Javier Luraschi (RStudio)
* Edward Ma (HP Enterprise)
* Luke Tierney (University of Iowa)
* Simon Urbanek (AT&T)

Minutes

Distributed Computing Working Group

2016-06-24T15:32:47Z

MichaelLawrence: Created page with "Goals and Purpose The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one op..."

Goals and Purpose

The Distributed Computing Working Group will endorse the design of a common abstraction for distributed data structures in R. We aim to have at least one open-source implementation, as well as a SQL implementation, released within a year of forming the group.

Members

'''Michael Lawrence''' (Genentech)
""Indrajit Roy"" (HP Enterprise)
''Joe Rickert'' (Microsoft)
Bernd Bischl ()
Matt Dowle (H2O)
Mario Inchiosa (Microsoft)
Michael Kane (Yale)
Javier Luraschi (RStudio)
Edward Ma (HP Enterprise)
Luke Tierney (University of Iowa)
Simon Urbanek (AT&T)

Minutes

Main Page

2016-06-24T15:22:57Z

MichaelLawrence:

== Welcome to the R Consortium Wiki ==

The R Consortium, Inc. is a group organized under an open source governance and foundation model to provide support to the R community, the R Foundation and groups and individuals, using, maintaining and distributing R software.

This wiki space is for working groups and ISC Project collaboration and documentation.

== Get Involved ==

* [https://www.r-consortium.org/ R Consortium Website]
* [https://www.r-consortium.org/about/isc/proposals Looking to Submit a Proposal]
* [https://twitter.com/RConsortium Follow us on Twitter]

== Working Groups ==

* [[R Native API|Native APIs for R]]
* [[Distributed Computing Working Group|Distributed Computing]]

'''''This wiki supports Linux Foundation ID single-sign-on and registration with the link at the top this page. Other components of the R Consortium that do not support single-sign-on will directly request your Linux Foundation ID username and password for login.'''''

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

Main Page

2016-06-24T15:20:11Z

MichaelLawrence:

== Welcome to the R Consortium Wiki ==

The R Consortium, Inc. is a group organized under an open source governance and foundation model to provide support to the R community, the R Foundation and groups and individuals, using, maintaining and distributing R software.

This wiki space is for working groups and ISC Project collaboration and documentation.

== Get Involved ==

* [https://www.r-consortium.org/ R Consortium Website]
* [https://www.r-consortium.org/about/isc/proposals Looking to Submit a Proposal]
* [https://twitter.com/RConsortium Follow us on Twitter]

== Working Groups ==

* [[R Native API|Native APIs for R]]
* [[Distributed Computing]]

'''''This wiki supports Linux Foundation ID single-sign-on and registration with the link at the top this page. Other components of the R Consortium that do not support single-sign-on will directly request your Linux Foundation ID username and password for login.'''''

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.