Distributed Computing Working Group - Revision history

Indrajit roy at 02:53, 12 May 2017

2017-05-12T02:53:20Z

2017-05-12T02:39:26Z

2017-05-12T02:38:53Z

2017-05-01T18:42:48Z

2017-05-01T17:56:12Z

2017-03-09T18:47:58Z

2016-12-09T18:48:40Z

Yuan works at Uptake

2016-12-09T05:27:29Z

‎12/08/2016

2016-12-09T05:26:16Z

‎Minutes

2016-11-10T22:42:50Z

‎11/10/2016

@@ Line 67: / Line 67: @@
 ** The plan() function can be used with futures to decide whether the futures are run in parallel or sequentially.
 ** The doFuture() package is a foreach adaptor. It allows foreach to utilize HPC clusters.
 === 03/09/2017 ===
 * Talk by Brian Lewis

@@ Line 58: / Line 58: @@
         f<- future(expo)
         v<-value(f)
 ** There are many ways to resolve futures. E.g., sequential, multicore, multisession, etc.
 ** Aim is to make the future package “write once, run anywhere”. It works on all platforms.

@@ Line 6: / Line 6: @@
 * '''Michael Lawrence''' (Genentech)
-* '''Indrajit Roy''' (HP Enterprise)
+* '''Indrajit Roy''' (Google)
 * ''Joe Rickert'' (ISC liason, RStudio)
 * Bernd Bischl (LMU)
@@ Line 51: / Line 51: @@
 == Minutes ==
 === 03/09/2017 ===
 * Talk by Brian Lewis

@@ Line 21: / Line 21: @@
 == Reports ==
-[[ddr2016|2016 Progress Report]]
+[[Distributed Computing Working Group Progress Report 2016|2016 Progress Report]]
 == Milestones ==

@@ Line 18: / Line 18: @@
 * Bryan Lewis (Paradigm4)
 * Hossein Falaki (databricks)
 == Milestones ==

@@ Line 47: / Line 47: @@
 == Minutes ==
 === 12/08/2016 ===
 * Yuan Tang from Uptake was the presenter

@@ Line 48: / Line 48: @@
 == Minutes ==
 === 12/08/2016 ===
-* Yuan Tang from RStudio was the presenter
+* Yuan Tang from Uptake was the presenter
 ** Michael and Indrajit will write a status report for the working group sometime in December or January
 ** Yuan gave an overview of TensorFlow

@@ Line 64: / Line 64: @@
 ** Yuan gave a brief overview of the python interface
 ** TensorFlow in R handles conversion between R and Python. Syntax is very similar to python API
 ** Future work: Adding more examples and tutorials, integration with Kubernetes/Marathon like framework.
 === 11/10/2016 ===

← Older revision		Revision as of 05:26, 9 December 2016
Line 47:		Line 47:

	== Minutes ==		== Minutes ==
		+	=== 12/08/2016 ===
		+	* Yuan Tang from RStudio was the presenter
		+	** Michael and Indrajit will write a status report for the working group sometime in December or January
		+	** Yuan gave an overview of TensorFlow
		+	** JJ, Dirk and Yuan are working on R layer for TensorFlow
		+	** TensorFlow is a platform for machine learning as well as other computations (even math proofs).
		+	** It is GPU optimized and distributed.
		+	** It is used in search, speech recognition, Google photos, etc.
		+	** TensorFlow computations are directed graphs. odes are operations and edges are tensors.
		+	** A lot of array, matrix, etc. operations are available
		+	** Backend is mostly C++. Python front end exists.
		+	** TensorFlow R is based on the python fronted
		+	** In multi-device setting, TensorFlow figures out which devices to use and manages communication between devices.
		+	** Computations are fault tolerant
		+	** Yuan has previously worked on Scikit Flow which is now TF.Learn. It’s a easy transition for Scikit learn users.
		+	** Yuan gave a brief overview of the python interface
		+	** TensorFlow in R handles conversion between R and Python. Syntax is very similar to python API
		+	** Future work: Adding more examples and tutorials, integration with Kubernetes/Marathon like framework.
	=== 11/10/2016 ===		=== 11/10/2016 ===

@@ Line 64: / Line 64: @@
 * Q/A
 ** Currently trying to get zero copy dataframe between python and Spark. Spark 2.0 has an off heap manager that uses Arrow. Once this feature is tested on the Python API, the next step will be integration R.
-** Spark dataframes gain from plan optimizations. It is not SparkR specific. R UDFs are still
+** Spark dataframes gain from plan optimizations. It is not SparkR specific. R UDFs are still treated as black boxes by the optimizer
-treated as black boxes by the optimizer
+** Spark doesn't directly support matrixes. There is no immediate intent to do so either. One can store an array or vector as a single column of a Spark dataframe.
 ** Spark doesn't directly support matrixes. There is no immediate intent to do so either.
 One can store an array or vector as a single column of a Spark dataframe.
 === 10/13/2016 ===