This image has only been tested for an AWS Glue 1.0 Spark shell (both for PySpark and Scala). In this post, we use amazon/aws-glue-libs:glue_libs_1.0.0_image_01 from Docker Hub.
#DOCKER FOR MAC QIITA CODE#
Also make sure that you have at least 7 GB of disk space for the image on the host running the Docker.įor more information about restrictions when developing AWS Glue code locally, see Local Development Restrictions. The machine running the Docker hosts the AWS Glue container. For installation instructions, see the Docker documentation for Mac, Windows, or Linux. Prerequisitesīefore you start, make sure that Docker is installed and the Docker daemon is running. We recommend pulling the highest image version for an AWS Glue major version to get the latest updates. So, the first image released for AWS Glue 2.0 will be glue_libs_2.0.0_image_01. Both these increments will be reset with every major AWS Glue release.
Image version will be incremented for the release of a new image of a major AWS Glue release.
#DOCKER FOR MAC QIITA PATCH#
The patch version will be incremented for updates to the AWS Glue libraries of a major release. 0 is the patch version, and 01 is the image version. In this name, 1.0 is the AWS Glue major version. The tags of the new images will follow the following convention: glue_libs_ _image_. The AWS Glue team will release new images for various AWS Glue updates. We have also bundled Jupyter and Zeppelin notebook servers in the image so you don’t have to configure an IDE and can start developing AWS Glue code right away. The image has AWS Glue 1.0, Apache Spark, OpenJDK, Maven, Python3, the AWS Command Line Interface (AWS CLI), and boto3. We have used these libraries to create an image with all the right dependencies packaged together. The AWS Glue team released the AWS Glue binaries and let you set up an environment on your desktop to test your code. AWS Glue comes with many improvements on top of Apache Spark and has its own ETL libraries that can fast-track the development process and reduce boilerplate code. AWS Glue is built on top of Apache Spark and therefore uses all the strengths of open-source technologies. In this post, we focus on writing ETL scripts for AWS Glue jobs locally. In the fourth post of the series, we discussed optimizing memory management. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load your data for analytics.
0 kommentar(er)
