Change directory command cd to D drive is not working in Command Prompt

In most cases, you can't change to a different drive using the cd command. The cd command only allows you to navigate within the current drive.

If you want to switch to a different drive in the command prompt, you need to specify the drive letter followed by a colon (:), for example:

d:

How Spark Job runs behind the scenes

1. The driver program divides the dataset into partitions and assigns each partition to an executor. 
2. Each executor initializes its memory and reads the data in its partition from the cluster. 
3. It then executes the tasks assigned to it, writing any intermediate output to disk. 
4. When the task is finished, the executor returns the result back to the driver program. 
5. The driver program then aggregates the results of the tasks and distributes them to the other nodes in the cluster.
6. Finally, the driver program collects the output and writes it to persistent storage.

How do you increase mappers in Hadoop

 To increase the number of mappers in Hadoop, you can increase the number of input splits by setting the mapreduce.input.fileinputformat.split.maxsize configuration parameter. Increasing this value can increase the number of mappers used by the application, but it can also lead to decreased performance if the splits are too large. The default value for mapreduce.input.fileinputformat.split.maxsize is 128 MB.

You can also tweak the size of each split and memory allocations for the application to optimize its performance.

Deprecated property: mapred.min.split.size

Scala vs Python (PySpark) with Scala, which is better?

Which is a better language for Spark programming, Data Engineering: Scala or Python?

It depends on your specific needs. Both Python and Scala are popular languages for data engineering and spark programming, but each have their own advantages. Python is often seen as the easier language to learn due to its simple syntax and intuitive design. Additionally, its extensive library of modules and open-source libraries make it useful for developing real-time applications. On the other hand, Scala is often seen as being more powerful and performant than Python due to its statically-typed nature and ability to easily integrate with Java. Ultimately, choosing the best language for data engineering depends on your individual requirements and preferences.

Is Scala still relevant?

Yes, Scala is still relevant. Although it has seen a decline in its popularity over the years, Scala is still actively used and maintained by many organizations. Its advantages, such as its ability to seamlessly integrate with Java and its functional programming capabilities, make it a valuable language in data engineering and analytics fields.

Why to use Pyspark instead of Scala with Spark?

PySpark is an interface for Python programming with Apache Spark and provides advantages such as allowing developers to write Python code in a Spark environment, as well as making it easier for those not familiar with Scala to work with Spark. Additionally, PySpark makes data processing faster, which can be useful for applications that require real-time analytics.

Why to use Scala with Spark instead of PySpark?

Scala is the most popular language for working with Apache Spark because of its ability to provide high performance and scalability. Scala also supports functional programming, which can be useful when dealing with large datasets as it allows developers to write concise and efficient code. Additionally, since Scala is a statically-typed language, it can be compiled more quickly, which leads to faster execution times.

Are data structures for Big Data / Data Engineering interviews easier in Python or Scala?

While both Python and Scala have their own advantages when it comes to programming data structures, many developers find Python to be slightly easier to work with due to its simpler syntax and more intuitive design. Additionally, since Python is a dynamically-typed language, it can be easier to write code for data structures quickly. However, Scala may be better suited for more complex data structures such as trees, graphs, or heaps, as it can offer more control over the structure and speed of data manipulation.

What is difference between Hive Metastore and MySQL Metastore?

Hive Metastore is a data storage layer that stores the structure of the data in a Hive warehouse. It acts as a central repository for metadata and also helps in providing access to external tools that need to access the Hive tables.

While MySQL Metastore is a type of database that is used to store Hive data such as table definitions, partitions, and statistics. Additionally, MySQL Metastore provides an interface to users to manage, query, and access the data stored in the Hive warehouse, while Hive Metastore does not have this capability.

How to run multiple queries at once in Toad?

To run multiple queries in Toad at once follow these steps:

1. Open Toad.
2. Go to View -> Toad Options. An Options pop up window will appear.
3. In the left pane click on: Oracle -> Transactions.
4. After clicking on Transactions in the left pane, on the right pane you will see a checkbox with "Execute queries in threads (Creates separate session)" checkbox.
5. By default it is unchecked, check it and click on Apply button on bottom right. Click on OK.

Now the next time you will connect to a database instance you will be able to run separate queries in different tabs without getting the 'Statement Processing' popup.

If the above post was helpful to you then please leave a message in the comments. Thanks.

Completely remove Java from Ubuntu 14.04

There maybe many reasons why you might want to remove Java from your system. Whatever the reasons, follow the procedure below to uninstall Java from your Ubuntu system:

Remove all the Java related packages

First run the below three commands one by one to remove all Java related packages like Sun, Oracle, OpenJDK, IcedTea plugins, GIJ:

Purge config files

After removing all the packages you need to purge all the config files. For that run the following command:

Remove cache directory

After purging the config files we need to remove Java config and cache directory from your system. Run the following command to remove the cache directory:

Remove manually installed JVM's

If you have installed any other JVM's then this is the time to remove them. Run the following command:

Remove all leftover Java entries

After uninstalling all the java components from your system some java entries might still be left in your system. To get rid of them run the following command:

Remove java directories from system

This is the final step in removing Java from your system. After uninstalling the java packages the java directories might still be there in your system. To remove them you need to first find if there are any left. Use the following commands to find any leftover directories:

If after running the above script you get any output similar to 'jre1.5/bin/pack200' then run the following command (replace jre1.5 with the directory name that you got in the output at the terminal) to remove the leftover java directory:

How to embed code snippets or files from github

Gist can help you embed code from Github into any of your posts.

How to use Gist to embed code from Github:

1. Go to Github. Login into your account.
2. Go to Gist.
3. Put your code into the box (copy and paste). You can optionally add Gist Description, the language you are using and the Gist name as well.
4. Click on create Public Gist at the bottom.
5. On the next page you will get the code for embedding the Gist that you just created. Copy it and paste it where ever you want to use this Gist.

Example gist:

Map Keys on Bluestacks - Play Clash of Clans efficiently on Bluestacks

The biggest headache while playing on Bluestacks is the delay in troop deployment due to no multitouch capabilities. Each time you need to drag your cursor to appropriate place in map to drop a troop and then again drag your cursor to switch to some other troop then take your cursor back to the location

Current troop deployment:
1. Drag your cursor to appropriate place in map to drop a troop (T1).
2. Drag your cursor to switch to a different troop (T2).
3. Again Drag your cursor to that appropriate place in map to drop T2.
4. Loop from 1 to 3 until battle ends.

How to fix this:
You can map keyboard keys in Bluestacks to tap on the screen at some predefined places(In this case the troop selection spot). Then you can just press 1 on keyboard to Select your troop 1, press 2 on keyboard to Select your troop 2 and so on for all troops.
Just train 9 types of Troops+Spells.
Then go to a farming attack.
Click the keyboard icon at the bottom to open settings to map keyboard keys. Then at the top right you will see the pencil icon, click that.
Now you will see Tap, D-Pad, Swipe, Tilt and Zoom options on top. Drag the Tap icon to where your first troop is and Map it as 1 (keyboard key). Do this again and again for all troops, you can see 11 types of Troop+Spells, so map 1-0 keys. I mapped P as the last one. These are the keyboard keys you will use to switch between troops.
Save.

Restart Bluestacks.

Open Clash of Clans. This time during an attack using 1 you will be able to switch to troop 1, using 2 you will be able to switch to troop 2 and so on.

Additionally you can also set swipe keys. I have set them as W,S,A and D.

Finally now my mapping looks like this:

jDeveloper display driver crashes while Scrolling - Display driver stopped responding and has recovered

If your display driver crashes while scrolling in jDeveloper and you get the message saying "Display driver stopped responding and has recovered" then this is the thread for you.

This can be fixed by modifying jdev.conf file. This file can be found at $JDEV_HOME/jdev/bin/jdev.conf or jdevbin\jdev\bin

Close jDeveloper. Make a backup of this file somewhere. Now go to the original file and add this line at the end of file:

AddVMOption -Dsun.java2d.noddraw=true

Save the file. Now start your jDeveloper (it may take 2-3 minutes for the first start after this). Scrolling should be fixed now.

If for some reason your jDeveloper does not starts then replace the original file back in jDeveloper.