Matthew and Jack interned with Novetta’s Machine Learning Center of Excellence during the summer of 2019. This blog series discusses ADSynth, an app that creates a digital architecture diagram from a photo of a whiteboard sketch.
As introduced in our first blog post, we built a simple iOS application alongside an AWS pipeline to provide an end-to-end solution to diagram creation.
AWS was the obvious solution to build a truly scalable system capable of supporting multiple users creating diagrams at once. We were already using GPU enabled EC2 instances during the development of our object detection model, hence most of the code we needed was already on an EC2 instance. However, we needed a way to make the model we developed and the rest of the necessary scripts accessible to the outside world such that an iOS application is able to interact with it.
Model Serving with Flask
To address this issue, we stood up a Flask server on our EC2 instance. The primary advantages of Flask are that it is extremely lightweight and can directly call and execute our ADSynth Python scripts. We configured an S3 bucket to serve as a repository for sending input pictures and receiving ADSynth output diagrams. A Lambda function is triggered whenever a new image is uploaded to the input folder of the S3 bucket. The function notifies the Flask server that an input image is ready for processing, along with the name and location of the image in the S3 bucket. On notification, the Flask server pulls the image from the bucket, creates the diagram, and uploads the new image back to the bucket.
In sum, the S3 bucket serves as the entry point to the diagram synthesis processes, and any client implementation is only responsible for sending the input images to S3 – everything else is handled automatically within AWS.
The iPhone App we designed is a simple, user-friendly client implementation that allows users to interact with ADSynth. We took a bottom-up approach and had our AWS backend built out and working before designing the user application. This approach enabled us to have a fully-functional app working in only a few hours. Since we were using OS X, we were able to create the iPhone application directly in Xcode. For Windows users, iOS applications can be created on Windows with React-Native.
In order to allow the iPhone application to send and receive data from the AWS bucket without making the S3 bucket public and open to the outside world, we created an AWS Cognito Identity and configured an IAM role with the appropriate level of permissions. With AWS Cognito, the application reaches out to the S3 bucket and verifies its authenticity as a trusted source using a secret key.
After taking a picture with the app, the photo is uploaded to S3 and the rest of the diagram synthesis process is automatically triggered. The app monitors the S3 bucket and waits for the diagram image to be uploaded for it to be retrieved and shown to the app user. Along with the JPEG version of the image (which is shown in the app), an editable SVG format is also uploaded to the S3 bucket. The workflow for getting the SVG image into a slide deck involves downloading the image directly from S3, but it is possible to automate this functionality depending on the use case. The app can easily be extended to include a registration process and email sign-in such that the SVG images are sent to the user’s email address.
And voilà, that’s it! A machine learning research project turned into a usable tool for Software Engineers and Solutions Architects.
That’s a Wrap!
Working on ADSynth was the highlight project during our summer internship at Novetta. Diving into the relatively unexplored problem space of synthesizing digital drawings proved to be challenging, yet taught us new technologies and forced us to think outside the box in order to come up with creative solutions. We were fortunate enough to work alongside and be mentored by machine learning researchers at Novetta. There is no limit as to what can be achieved with object detection models such as YOLOv3 and it will be interesting to see how it is applied to other projects in the future. We’re excited to see what the next intern class accomplishes!
Read the series!
See the demo at Strata Data Conference!
Visit Novetta at booth 1201 to speak with our subject matter experts and see a live demo of ADSynth.
Strata Data Conference • New York
September 23-26, 2019