Top 5 mistakes to avoid when writing Apache Spark applications

-

Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Spark is one of the big data engines that are trending in recent times. One of the main reasons is that is because of its ability to process real-time streaming data. Its advantages over traditional MapReduce are:
  • It is faster than MapReduce
  • Well equipped with Machine Learning abilities.
  • Supports multiple programming languages.
However, in spite of having all these advantages over Hadoop, we often get stuck in certain situations which arise due to inefficient codes are written for applications. The situations and their solutions are discussed below:
  • Always try to use reducebykey instead of groupbykey
  • Reduce should be lesser than TreeReduce
  • Always try to lower the side of maps as much as possible
  • Try not to shuffle more
  • Try to keep away from Skews as well as partitions too

Do not let the jobs to slow down:

When the application is shuffled, it takes more time around 4 long hours to run. This makes the system slow.
There are two stages of aggregation and they are:
  • action on the salted keys
  • action on the unsalted keys
So we have to remove the isolated keys and then accumulation should be used which will decrease the data used as a result we can huge information can be saved from being shuffled.

Avoid wrong dimensions of executors

In any particular Spark jobs, executors are the executing nodes that are responsible for processing singular tasks in the job. These executors provide in-memory storage for RDDs that are cached by user programs through Block Manager. They are created at the very starting of the particular Spark application and are on for the whole application span. After processing the entity works, the deliver the output to the driver. The mistakes that we do during the writing of the Spark application with the executors are that we take the wrong size executors. Things that we go wrong in the assigning of the following:
  • Number of Executors
  • Cores of each executor
  • Memory for each executor
Normally people use 6 executors, 16 cores each and 64 GB of RAM.
When using 16 core for each executor, the total number of cores for 16 executors become 96. And the memory per node becomes 64/16 i.e. 4 GB for each executor. Hence if it becomes most granular for using smallest size executors we fail to make use of the advantages of processing all the tasks in the same java virtual machine. But in the same calculation if it becomes least granular also it becomes a problem because no memory remains free for overhead for OS/Hadoop daemons. And instead of 16 cores, if we use 15 cores also it will result in bad throughput. So the perfect number is 5 cores per executor. Because for 6*15 we have 90 cores, so the number of executors will be 90/5 i.e 18. So leaving one executor for AM, we have 17 remaining, so executors in 1 node will be 3, Hence, RAM= 63/3= 21 GB, 21 x (1–0.07) ~ 19 GB. Therefore for correct application people should use 17 executors, 5 cores each and 19 GB of RAM.
The above tips should be followed in order to avoid mistakes while writing an Apache Spark Application.
Thanks for dropping by !!! Feel free to comment to this post or you can also send me an email at naik899@gmail.com
The post Top 5 mistakes to avoid when writing Apache Spark applicationsappeared first on RavindraNaik.

Comments

Post a Comment

Popular posts from this blog

Serve GZip CSS and JS from AWS S3

-
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. If you are looking to gzip  your static JS and CSS when hosting on S3, however, it’s not obvious how to do so, and results often bring up other AWS hosting services as well. Here’s the quick and simple way to serve gzipped JS and CSS from AWS S3. S3 needs those files compressed before they are uploaded, it can’t do the actual gzipping for us. This is done with a simple command, which I recommend be added as a script to your build step, or your continuous integration process. gzip -9 /filepath/bootstrap.js or gzip -9 /filepath/bootstrap.css The -9 denotes that we want the
Read more

Create a Central Logging System

-
Image
Logging is the most important part of any system. They give you insights about the application, what kind of errors are occurring and what components are causing the errors and also about how the application is reacting and working when something wrong happens. The usual practice is almost every application write to logs either in the file system or in Database. If your application is running on multiple hosts then designing a central logging system becomes crucial since you can collect, aggregate and maintain logs at one centralized location and do operations on top of them. There are many tools available to which can solve some part of the problem but we need to build a robust application using all these tools.                                                                           Create a Centralised Logging Application Collection Imagine you have a web application running on the server and if something goes down, your developers or operations team need to access lo
Read more

Create a calendar invite (ics file) in ASP NET

-
Creating a calendar invite (ics file) and attaching a email via service or sending an email with downloadable calendar invite is what we are going to learn in following posts. In this post, I will show how you can use ICal library from Nuget and create a calendar invite. The calendar begins with the keyword  BEGIN:VCALENDAR  and ends with  END:VCALENDAR . In between the calendar you can add as many events as you want beginning each event with  BEGIN:VEVENT  and ending with  END:VEVENT . There is a catch when adding multiple events to a calendar, and that is the type of METHOD you use. With METHOD:PUBLISH, you can add multiple events and upon opening the iCal with MS Outlook it will create a new calendar, separate from your main Exchange calendar. METHOD:REQUEST creates a calendar invite, so you can only include one event and upon opening with MS Outlook it will ask you to accept the event and add it to your main Exchange calendar. With this method you have to include the organizer
Read more