Download as PDF, PPTX
![Pipeline
p = beam.Pipeline('DirectRunner')
(p | 'input' >> beam.Create(['Hello', 'World'])
| 'output' >> beam.io.WriteToText('gs://bucket/hello')
)
p.run()
(Pythonista )](https://image.slidesharecdn.com/dataflowdetensorflow-170315145018/85/Google-Cloud-Dataflow-meets-TensorFlow-13-320.jpg)
![Pipeline
(p | 'input' >> beam.Create(['Hello', 'World'])
| 'output' >> beam.io.WriteToText('gs://bucket/hello')
)
Pipeline |](https://image.slidesharecdn.com/dataflowdetensorflow-170315145018/85/Google-Cloud-Dataflow-meets-TensorFlow-14-320.jpg)
![PCollection
beam.Create(['Hello', 'World'])
plaintext 'Hello' 'World' 2
In Memory](https://image.slidesharecdn.com/dataflowdetensorflow-170315145018/85/Google-Cloud-Dataflow-meets-TensorFlow-17-320.jpg)
![train
TensorFlow OK Dataflow
def train(param):
import tensorflow as tf
from sklearn import cross_validation
#
iris = tf.contrib.learn.datasets.base.load_iris()
train_x, test_x, train_y, test_y = cross_validation.train_test_split(
iris.data, iris.target, test_size=0.2, random_state=0
)
# https://www.tensorflow.org/get_started/tflearn
feature_columns = [tf.contrib.layers.real_valued_column("", dimension=4)]
classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
hidden_units=param['hidden_units'],
dropout=param['dropout'],
n_classes=3,
model_dir='gs://{BUCKET}/models/%s'% model_id)
classifier.fit(x=train_x,
y=train_y,
steps=param['steps'],
batch_size=50)
result = classifier.evaluate(x=test_x, y=test_y)
ret = {'accuracy': float(result['accuracy']),
'loss': float(result['loss']),
'model_id': model_id,
'param': json.dumps(param)}
return ret](https://image.slidesharecdn.com/dataflowdetensorflow-170315145018/85/Google-Cloud-Dataflow-meets-TensorFlow-29-320.jpg)
![Protobuf Datastore Client lib
def csv_format(entity_pb):
entity = entity_from_protobuf(entity_pb)
columns = ['"%s"' % entity[k]
for k in sorted(entity.keys())]
return ','.join(columns)
p = beam.Pipeline(options=options)
(p | 'read from datastore' >>
ReadFromDatastore(project=PROJECTID,
query=query)
| 'format entity to csv' >>
beam.Map(csv_format)
...](https://image.slidesharecdn.com/dataflowdetensorflow-170315145018/85/Google-Cloud-Dataflow-meets-TensorFlow-57-320.jpg)
Uploaded byHayato Yoshikawa
Google Cloud Dataflow meets TensorFlow
Google Cloud Dataflow can be used to build TensorFlow pipelines. Dataflow allows training multiple TensorFlow models in parallel and writing results to Cloud Datastore. A sample pipeline shows generating training parameters, mapping over them to train models, and writing accuracy results to Cloud Storage. Dataflow provides autoscaling and machine types can be configured. The new DatastoreIO allows reading from and writing to Cloud Datastore from Dataflow pipelines using Protobuf and entity conversion helpers.
More Related Content
Similar to Google Cloud Dataflow meets TensorFlow
Google Cloud Dataflow meets TensorFlow
- 1. Google Cloud Dataflow meetsTensorFlow Dataflow TensorFlow Datastore H.Yoshikawa (@hayatoy) GCPUG Shonan #12 25 Mar. 2017
- 2.
- 3.
- 5.
- 6.
- 7. Google Cloud Dataflow MapReduce GCE JupyterNotebook deploy
- 8.
- 9. Prerequisites Create GCP account Enablebilling Enable Google Dataflow API
- 10. Installation Apache Beam $ gitclone https://github.com/apache/beam.git $ cd beam/sdks/python/ $ python setup.py sdist $ cd dist/ $ pip install apache-beam-sdk-*.tar.gz >> 0.7.0.dev0 google‑cloud‑dataflow $ pip install google-cloud-dataflow >> 0.5.5
- 11. Installation # gcloud-python $ pipinstall gcloud >> 0.18.3 # Application Default Credentials $ gcloud beta auth application-default login gs://
- 12.
- 13. Pipeline p = beam.Pipeline('DirectRunner') (p| 'input' >> beam.Create(['Hello', 'World']) | 'output' >> beam.io.WriteToText('gs://bucket/hello') ) p.run() (Pythonista )
- 14. Pipeline (p | 'input'>> beam.Create(['Hello', 'World']) | 'output' >> beam.io.WriteToText('gs://bucket/hello') ) Pipeline |
- 15.
- 16.
- 17. PCollection beam.Create(['Hello', 'World']) plaintext 'Hello''World' 2 In Memory
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. Q: Google CloudDataflow TensorFlow A:
- 26. TensorFlow Pipeline (p |'generate params' >> beam.Create(params) | 'train' >> beam.Map(train) | 'output' >> beam.io.WriteToText('gs://bucket/acc') ) generate params train output
- 28. train # (p |'generate params' >> beam.Create(params) | 'train' >> beam.Map(train) # ← # | 'output' >> beam.io.WriteToText('gs://bucket/acc') # ) PCollection PCollection Windowing Worker Prediction PCollection
- 29. train TensorFlow OK Dataflow deftrain(param): import tensorflow as tf from sklearn import cross_validation # iris = tf.contrib.learn.datasets.base.load_iris() train_x, test_x, train_y, test_y = cross_validation.train_test_split( iris.data, iris.target, test_size=0.2, random_state=0 ) # https://www.tensorflow.org/get_started/tflearn feature_columns = [tf.contrib.layers.real_valued_column("", dimension=4)] classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns, hidden_units=param['hidden_units'], dropout=param['dropout'], n_classes=3, model_dir='gs://{BUCKET}/models/%s'% model_id) classifier.fit(x=train_x, y=train_y, steps=param['steps'], batch_size=50) result = classifier.evaluate(x=test_x, y=test_y) ret = {'accuracy': float(result['accuracy']), 'loss': float(result['loss']), 'model_id': model_id, 'param': json.dumps(param)} return ret
- 30.
- 33.
- 34. Pipeline (p | 'generateparams' >> beam.Create(params) | 'train' >> beam.Map(train) | 'output' >> beam.io.WriteToText('gs://bucket/acc') )
- 37. (p | 'generateparams' >> beam.Create(params) # | 'train' >> beam.Map(train) # | 'output' >> beam.io.WriteToText('gs://bucket/acc') #) PCollection train Dataflow
- 38.
- 39. worker_options.max_num_workers = 10 worker_options.num_workers= 10 worker_options.disk_size_gb = 20 worker_options.machine_type = 'n1-standard-16' max_num_workers num_workers auto scaling disk_size_gb GB 250GB machine_type GCE REST GPU
- 41.
- 43.
- 44.
- 45. Branch def split_branch(n, side=0): ifn % 2 == side: yield n pipe_0 = p | 'param' >> beam.Create(range(100)) branch1 = (pipe_0 | 'branch1' >> beam.FlatMap(split_branch, 0)) branch2 = (pipe_0 | 'branch2' >> beam.FlatMap(split_branch, 1))
- 46. Dynamic Work Rebalancing Strugglewith Stragglers
- 48. 2000 20 → 5/ → 5 DNN 15 ON/OFF OFF
- 49.
- 50.
- 51.
- 52.
- 53. DatastoreIO (Python ) PythonDataflow GA DatastoreIO Protobuf ← New!
- 54. from apache_beam.io.gcp.datastore.v1.datastoreio importReadFromDatastore from apache_beam.io.gcp.datastore.v1.datastoreio import WriteToDatastore from google.cloud.proto.datastore.v1 import entity_pb2 from google.cloud.proto.datastore.v1 import query_pb2 from googledatastore import helper as datastore_helper, PropertyFilter from gcloud.datastore.helpers import entity_from_protobuf DatastoreSource DatastoreSink → (Beam v0.7.0 )
- 55. Read from Datastore Pipeline (p| 'read from datastore' >> ReadFromDatastore(project=PROJECTID, query=query) ...
- 56. Query query = query_pb2.Query() query.kind.add().name= 'Test' datastore_helper.set_property_filter( query.filter, 'foo', PropertyFilter.EQUAL, 'lorem' ) GQL SELECT * FROM Test WHERE foo = 'lorem'
- 57. Protobuf Datastore Clientlib def csv_format(entity_pb): entity = entity_from_protobuf(entity_pb) columns = ['"%s"' % entity[k] for k in sorted(entity.keys())] return ','.join(columns) p = beam.Pipeline(options=options) (p | 'read from datastore' >> ReadFromDatastore(project=PROJECTID, query=query) | 'format entity to csv' >> beam.Map(csv_format) ...
- 58. Write to Datastore Pipeline ... |'create entity' >> beam.Map(create_entity) | 'write to datastore' >> WriteToDatastore(project=PROJECTID))
- 59. Entity Protobuf def create_entity(param): entity =entity_pb2.Entity() datastore_helper.add_key_path(entity.key, 'Test', str(uuid.uuid4())) datastore_helper.add_properties(entity, {"foo": u"hoge", "bar": u"fuga", "baz": 42}) return entity
- 60.
- 61.
- 62.