Hadoop Interview Questions - Set 4

What is Hive ?

Hive is a data warehouse software which is used for facilitating querying and managing large data sets residing in distributed storage.Hive language almost look like SQL language called HiveQL.Hive also allows traditional map reduce programs to customize mappers and reducers when it is inconvenient or inefficient to execute the logic in HiveQL (User Defined Functions UDFS)

What is Hive Metastore ?

Hive Meta store is a database that stores metadata of your hive tables like table name,column name,data types,table location,number of buckets in the table etc.

What is the present version of Hive?

HIVE-0.13.1

What is the stable version of Hive ?

HIVE-0.12.0

Hive new version supported Hadoop Versions ?

This release works with Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y

Where we have to set the Hive Installation Path ?

we can set hive path in ~/.bashrc file or hadoop-env.sh file
~/.bashrc file or hadoop-env.sh file which one is better to set the path ?
~/.bashrc.sh

Why ~/.bashrc.sh is better then hadoop-env.sh ?

~/.bashrc.sh starts the work when system login but hadoop-env.sh starts the work only when hadoop starts in system

What is Hive Installation Path ?

export HIVE_HOME=/home/hadoop/work/hive-x.y.z
export PATH=$PATH:$HIVE_HOME/bin

How to Install Hive ?

check the answer above menu-hive installation tab

Which companies are mostly using Hive ?

Facebook,Netflix

Which company initially developed Hive ?

Facebook

How Facebook Uses Hadoop,Hive and Hbase ?

Facebook data stored on HDFS,everyday millions of photos uploaded into facebook with the help of Hadoop
Facebook Messages,Likes and statues updates running on top of HbaseHive to generate reports for third-party developers and advertisers who need to track the success of their applications or campaigns.What is Apache Hcatalog ?
HCatalog is built on top of the Hive metastore and incorporates Hive’s DDL.Apache Hcatalog is a table and data management layer for hadoop,we can process the data on Hcatalog by using APache pig,Apache Mapreduce and Apache Hive.There is no need to worry in Hcatalog where data is stored and which format of data generated.HCatalog displays data from RCFile format, text files, or sequence files in a tabular view. It also provides REST APIs so that external systems can access these tables’ metadata.

What is the work of Hive/Hcatalog ?

Hive/HCatalog also enables sharing of data structure with external systems including traditional data management tools.

What is WebHCatServer ?

The WebHcatServer provides a REST – like web API for Hcatalog.Applications make HTTP requests to run Pig, Hive, and HCatalog DDL from within applications.

What is SerDe in Apache Hive ?

SerDe full form is  Serializer Deserializer.Hive uses  Serializer Deserializer to read and write the data from hive table.The importent one behind hive is hive does not have own Hadoop distributed file system(HDFS) format that data is stored in.Users have to write store the hive data on HDFS by using (“CREATE EXTERNAL TABLE” or “LOAD DATA INPATH,” ) and use Hive to correctly “parse” that file format in a way that can be used by Hive.Hive uses to “parse” data stored in HDFS to be used by Hive

Is it possible to use same metastore by multiple users, in case of embedded hive?

No, it is not possible to use metastore in sharing mode. It is recommended to use standalone “real” database like MySQL or PostGresSQL.

Is multiline comment supported in Hive Script ?

NO

Difference between SQL and HiveQL ?

Hive Data types ?

Hadoop Interview Questions - Set 3

Hadoop Interview Questions (Set-3)

What is a JobTracker in Hadoop? How many instances of JobTracker run on a Hadoop Cluster?

JobTracker is the daemon service for submitting and tracking MapReduce jobs in Hadoop. There is only One Job Tracker process run on any hadoop cluster. Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine. Each slave node is configured with job tracker node location. The JobTracker is single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. JobTracker in Hadoop performs following actions(from Hadoop Wiki:)

• Client applications submit jobs to the Job tracker.
• The JobTracker talks to the NameNode to determine the location of the data
• The JobTracker locates TaskTracker nodes with available slots at or near the data
• The JobTracker submits the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.
• A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.
• When the work is completed, the JobTracker updates its status
• Client applications can poll the JobTracker for information.

How JobTracker schedules a task?

The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated. When the JobTracker tries to find somewhere to schedule a task within the MapReduce operations, it first looks for an empty slot on the same server that hosts the DataNode containing the data, and if not, it looks for an empty slot on a machine in the same rack.

What is a Task Tracker in Hadoop? How many instances of TaskTracker run on a Hadoop Cluster

A TaskTracker is a slave node daemon in the cluster that accepts tasks (Map, Reduce and Shuffle operations) from a JobTracker. There is only One Task Tracker process run on any hadoop slave node. Task Tracker runs on its own JVM process. Every TaskTracker is configured with a set of slots, these indicate the number of tasks that it can accept. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these task instances, capturing the output and exit codes. When the Task instances finish, successfully or not, the task tracker notifies the JobTracker. The TaskTrackers also send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated.

What is a Task instance in Hadoop? Where does it run?

Task instances are the actual MapReduce jobs which are run on each slave node. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. Each Task Instance runs on its own JVM process. There can be multiple processes of task instance running on a slave node. This is based on the number of slots configured on task tracker. By default a new task instance JVM process is spawned for a task.

How many Daemon processes run on a Hadoop system?

Hadoop is comprised of five separate daemons. Each of these daemon run in its own JVM.Following 3 Daemons run on Master nodes NameNode - This daemon stores and maintains the metadata for HDFS. Secondary NameNode - Performs housekeeping functions for the NameNode. JobTracker - Manages MapReduce jobs, distributes individual tasks to machines running the Task Tracker. Following 2 Daemons run on each Slave nodes DataNode – Stores actual HDFS data blocks. TaskTracker - Responsible for instantiating and monitoring individual Map and Reduce tasks.

What is configuration of a typical slave node on Hadoop cluster? How many JVMs run on a slave node?

• Single instance of a Task Tracker is run on each Slave node. Task tracker is run as a separate JVM process.
• Single instance of a DataNode daemon is run on each Slave node. DataNode daemon is run as a separate JVM process.
• One or Multiple instances of Task Instance is run on each slave node. Each task instance is run as a separate JVM process. The number of Task instances can be controlled by configuration. Typically a high end machine is configured to run more task instances.

What is the difference between HDFS and NAS ?

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. Following are differences between HDFS and NAS

• In HDFS Data Blocks are distributed across local drives of all machines in a cluster. Whereas in NAS data is stored on dedicated hardware.
• HDFS is designed to work with MapReduce System, since computation are moved to data. NAS is not suitable for MapReduce since data is stored seperately from the computations.
• HDFS runs on a cluster of machines and provides redundancy usinga replication protocal. Whereas NAS is provided by a single machine therefore does not provide data redundancy.

How NameNode Handles data node failures?

NameNode periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode. When NameNode notices that it has not recieved a hearbeat message from a data node after a certain amount of time, the data node is marked as dead. Since blocks will be under replicated the system begins replicating the blocks that were stored on the dead datanode. The NameNode Orchestrates the replication of data blocks from one datanode to another. The replication data transfer happens directly between datanodes and the data never passes through the namenode.

Does MapReduce programming model provide a way for reducers to communicate with each other? In a MapReduce job can a reducer communicate with another reducer?

Nope, MapReduce programming model does not allow reducers to communicate with each other. Reducers run in isolation.

Can I set the number of reducers to zero?

Yes, Setting the number of reducers to zero is a valid configuration in Hadoop. When you set the reducers to zero no reducers will be executed, and the output of each mapper will be stored to a separate file on HDFS. [This is different from the condition when reducers are set to a number greater than zero and the Mappers output (intermediate data) is written to the Local file system(NOT HDFS) of each mappter slave node.]

Where is the Mapper Output (intermediate kay-value data) stored ?

The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.

What are combiners? When should I use a combiner in my MapReduce Job?

Combiners are used to increase the efficiency of a MapReduce program. They are used to aggregate intermediate map output locally on individual mapper outputs. Combiners can help you reduce the amount of data that needs to be transferred across to the reducers. You can use your reducer code as a combiner if the operation performed is commutative and associative. The execution of combiner is not guaranteed, Hadoop may or may not execute a combiner. Also, if required it may execute it more then 1 times. Therefore your MapReduce jobs should not depend on the combiners execution.

What is Writable & WritableComparable interface?

• org.apache.hadoop.io.Writable is a Java interface. Any key or value type in the Hadoop Map-Reduce framework implements this interface. Implementations typically implement a static read(DataInput) method which constructs a new instance, calls readFields(DataInput) and returns the instance.
• org.apache.hadoop.io.WritableComparable is a Java interface. Any type which is to be used as a key in the Hadoop Map-Reduce framework should implement this interface. WritableComparable objects can be compared to each other using Comparators.

What is the Hadoop MapReduce API contract for a key and value Class?

• The Key must implement the org.apache.hadoop.io.WritableComparable interface.
• The value must implement the org.apache.hadoop.io.Writable interface.

What is a IdentityMapper and IdentityReducer in MapReduce ?

• org.apache.hadoop.mapred.lib.IdentityMapper Implements the identity function, mapping inputs directly to outputs. If MapReduce programmer do not set the Mapper Class using JobConf.setMapperClass then IdentityMapper.class is used as a default value.
• org.apache.hadoop.mapred.lib.IdentityReducer Performs no reduction, writing all input values directly to the output. If MapReduce programmer do not set the Reducer Class using JobConf.setReducerClass then IdentityReducer.class is used as a default value.

What is the meaning of speculative execution in Hadoop? Why is it important?

Speculative execution is a way of coping with individual Machine performance. In large clusters where hundreds or thousands of machines are involved there may be machines which are not performing as fast as others. This may result in delays in a full job due to only one machine not performaing well. To avoid this, speculative execution in hadoop can run multiple copies of same map or reduce task on different slave nodes. The results from first node to finish are used.

When is the reducers are started in a MapReduce job?

In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.

If reducers do not start before all mappers finish then why does the progress on MapReduce job shows something like Map(50%) Reduce(10%)? Why reducers progress percentage is displayed when mapper is not finished yet?

Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The progress calculation also takes in account the processing of data transfer which is done by reduce process, therefore the reduce progress starts showing up as soon as any intermediate key-value pair for a mapper is available to be transferred to reducer. Though the reducer progress is updated still the programmer defined reduce method is called only after all the mappers have finished.

What is HDFS ? How it is different from traditional file systems?

HDFS, the Hadoop Distributed File System, is responsible for storing huge data on the cluster. This is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant.

• HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
• HDFS provides high throughput access to application data and is suitable for applications that have large data sets.
• HDFS is designed to support very large files. Applications that are compatible with HDFS are those that deal with large data sets. These applications write their data only once but they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS supports write-once-read-many semantics on files.

What is HDFS Block size? How is it different from traditional file system block size?

In HDFS data is split into blocks and distributed across multiple nodes in the cluster. Each block is typically 64Mb or 128Mb in size. Each block is replicated multiple times. Default is to replicate each block three times. Replicas are stored on different nodes. HDFS utilizes the local file system to store each HDFS block as a separate file. HDFS Block size can not be compared with the traditional file system block size.

What is a NameNode? How many instances of NameNode run on a Hadoop Cluster?

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself. There is only One NameNode process run on any hadoop cluster. NameNode runs on its own JVM process. In a typical production cluster its run on a separate machine. The NameNode is a Single Point of Failure for the HDFS Cluster. When the NameNode goes down, the file system goes offline. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives.

What is a DataNode? How many instances of DataNode run on a Hadoop Cluster?

A DataNode stores data in the Hadoop File System HDFS. There is only One DataNode process run on any hadoop slave node. DataNode runs on its own JVM process. On startup, a DataNode connects to the NameNode. DataNode instances can talk to each other, this is mostly during replicating data.

How the Client communicates with HDFS?

The Client communication to HDFS happens using Hadoop HDFS API. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file on HDFS. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data.

How the HDFS Blocks are replicated?

HDFS is designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are write-once and have strictly one writer at any time. The NameNode makes all decisions regarding replication of blocks. HDFS uses rack-aware replica placement policy. In default configuration there are total 3 copies of a datablock on HDFS, 2 copies are stored on datanodes on same rack and 3rd copy on a different rack.

Can you think of a questions which is not part of this post? Please don't forget to share it with me in comments section & I will try to include it in the list.

Hadoop Interview Questions - Set 2

Hadoop Interview Questions (Set-2)

1) What is Hadoop Map Reduce ?

For processing large data sets in parallel across a hadoop cluster, Hadoop MapReduce framework is used.  Data analysis uses a two-step map and reduce process.

2) How Hadoop MapReduce works?

In MapReduce, during the map phase it counts the words in each document, while in the reduce phase it aggregates the data as per the document spanning the entire collection. During the map phase the input data is divided into splits for analysis by map tasks running in parallel across Hadoop framework.

3) Explain what is shuffling in MapReduce ?

The process by which the system performs the sort and transfers the map outputs to the reducer as inputs is known as the shuffle

4) Explain what is distributed Cache in MapReduce Framework ?

Distributed Cache is an important feature provided by map reduce framework. When you want to share some files across all nodes in Hadoop Cluster, DistributedCache  is used.  The files could be an executable jar files or simple properties file.

5) Explain what is NameNode in Hadoop?

NameNode in Hadoop is the node, where Hadoop stores all the file location information in HDFS (Hadoop Distributed File System).  In other words, NameNode is the centrepiece of an HDFS file system.  It keeps the record of all the files in the file system, and tracks the file data across the cluster or multiple machines

6) Explain what is JobTracker in Hadoop? What are the actions followed by Hadoop?

In Hadoop for submitting and tracking MapReduce jobs,  JobTracker is used. Job tracker run on its own JVM process

Hadoop performs following actions in Hadoop

• Client application submit jobs to the job tracker
• JobTracker communicates to the Namemode to determine data location
• Near the data or with available slots JobTracker locates TaskTracker nodes
• On chosen TaskTracker Nodes, it submits the work
• When a task fails, Job tracker notify and decides what to do then.
• The TaskTracker nodes are monitored by JobTracker

7) Explain what is heartbeat in HDFS?

Heartbeat is referred to a signal used between a data node and Name node, and between task tracker and job tracker, if the Name node or job tracker does not respond to the signal, then it is considered there is some issues with data node or task tracker

8) Explain what combiners is and when you should use a combiner in a MapReduce Job?

To increase the efficiency of MapReduce Program, Combiners are used.  The amount of data can be reduced with the help of combiner’s that need to be transferred across to the reducers. If the operation performed is commutative and associative you can use your reducer code as a combiner.  The execution of combiner is not guaranteed in Hadoop

9) What happens when a datanode fails ?

When a datanode fails

• Jobtracker and namenode detect the failure
• On the failed node all tasks are re-scheduled
• Namenode replicates the users data to another node

10) Explain what is Speculative Execution?

In Hadoop during Speculative Execution a certain number of duplicate tasks are launched.  On different slave node, multiple copies of same map or reduce task can be executed using Speculative Execution. In simple words, if a particular drive is taking long time to complete a task, Hadoop will create a duplicate task on another disk.  Disk that finish the task first are retained and disks that do not finish first are killed.

11) Explain what are the basic parameters of a Mapper?

The basic parameters of a Mapper are

• LongWritable and Text
• Text and IntWritable

12) Explain what is the function of MapReducer partitioner?

The function of MapReducer partitioner is to make sure that all the value of a single key goes to the same reducer, eventually which helps evenly distribution of the map output over the reducers

13) Explain what is difference between an Input Split and HDFS Block?

Logical division of data is known as Split while physical division of data is known as HDFS Block

14) Explain what happens in textinformat ?

In textinputformat, each line in the text file is a record.  Value is the content of the line while Key is the byte offset of the line. For instance, Key: longWritable, Value: text

15) Mention what are the main configuration parameters that user need to specify to run Mapreduce Job ?

The user of Mapreduce framework needs to specify

• Job’s input locations in the distributed file system
• Job’s output location in the distributed file system
• Input format
• Output format
• Class containing the map function
• Class containing the reduce function
• JAR file containing the mapper, reducer and driver classes

16) Explain what is WebDAV in Hadoop?

To support editing and updating files WebDAV is a set of extensions to HTTP.  On most operating system WebDAV shares can be mounted as filesystems , so it is possible to access HDFS as a standard filesystem by exposing HDFS over WebDAV.

17)  Explain what is sqoop in Hadoop ?

To transfer the data between Relational database management (RDBMS) and Hadoop HDFS a tool is used known as Sqoop. Using Sqoop data can be transferred from RDMS like MySQL or Oracle into HDFS as well as exporting data from HDFS file to RDBMS

18) Explain how JobTracker schedules a task ?

The task tracker send out heartbeat messages to Jobtracker usually every few minutes to make sure that JobTracker is active and functioning.  The message also informs JobTracker about the number of available slots, so the JobTracker can stay upto date with where in the cluster work can be delegated

19) Explain what is Sequencefileinputformat?

Sequencefileinputformat is used for reading files in sequence. It is a specific compressed binary file format which is optimized for passing data between the output of one MapReduce job to the input of some other MapReduce job.

20) Explain what does the conf.setMapper Class do ?

Conf.setMapperclass  sets the mapper class and all the stuff related to map job such as reading data and generating a key-value pair out of the mapper.

Hadoop Interview Questions (Set-1)

There is given hadoop interview questions and answers that has been asked in many companies. Let's see the list of top hadoop interview questions.

---

1) What is Hadoop?

Hadoop is a distributed computing platform. It is written in Java. It consist of the features like Google File System and MapReduce.

---

2) What platform and Java version is required to run Hadoop?

Java 1.6.x or higher version are good for Hadoop, preferably from Sun. Linux and Windows are the supported operating system for Hadoop, but BSD, Mac OS/X and Solaris are more famous to work.

---

3) What kind of Hardware is best for Hadoop?

Hadoop can run on a dual processor/ dual core machines with 4-8 GB RAM using ECC memory. It depends on the workflow needs.

---

4) What are the most common input formats defined in Hadoop?

These are the most common input formats defined in Hadoop:

1. TextInputFormat
2. KeyValueInputFormat
3. SequenceFileInputFormat

TextInputFormat is a by default input format.

---

5) What is InputSplit in Hadoop? Explain.

When a hadoop job runs, it splits input files into chunks and assign each split to a mapper for processing. It is called InputSplit.

---

6) How many InputSplits is made by a Hadoop Framework?

Hadoop will make 5 splits as following:

• One split for 64K files
• Two splits for 65MB files, and
• Two splits for 127MB files

---

7) What is the use of RecordReader in Hadoop?

InputSplit is assigned with a work but doesn't know how to access it. The record holder class is totally responsible for loading the data from its source and convert it into keys pair suitable for reading by the Mapper. The RecordReader's instance can be defined by the Input Format.

---

8) What is JobTracer in Hadoop?

is a service within Hadoop which runs MapReduce jobs on the cluster.

---

9) What are the functionalities of JobTracer?

These are the main tasks of JobTracer:

• To accept jobs from client.
• To communicate with the NameNode to determine the location of the data.
• To locate TaskTracker Nodes with available slots.
• To submit the work to the chosen TaskTracker node and monitors progress of each tasks.

---

10) Define TaskTracker.

TaskTracker is a node in the cluster that accepts tasks like MapReduce and Shuffle operations from a JobTracker.

---

11) What is Map/Reduce job in Hadoop?

Map/Reduce is programming paradigm which is used to allow massive scalability across the thousands of server.

Actually MapReduce refers two different and distinct tasks that Hadoop performs. In the first step maps jobs which takes the set of data and converts it into another set of data and in the second step, Reduce job. It takes the output from the map as input and compress those data tuples into smaller set of tuples.

---

12) What is Hadoop Streaming?

Hadoop streaming is a utility which allows you to create and run map/reduce job. It is a generic API that allows programs written in any languages to be used as Hadoop mapper.

---

13) What is a combiner in Hadoop?

A Combiner is a mini-reduce process which operates only on data generated by a Mapper. When Mapper emits the data, combiner receives it as input and sends the output to reducer.

---

14) Is it necessary to know java to learn Hadoop?

If you have a background in any programming language like C, C++, PHP, Python, Java etc. It may be really helpful, but if you are nil in java, it is necessary to learn Java and also get the basic knowledge of SQL.

---

15) How to debug Hadoop code?

There are many ways to debug Hadoop codes but the most popular methods are:

• By using Counters.
• By web interface provided by Hadoop framework.

---

16) Is it possible to provide multiple inputs to Hadoop? If yes, explain.

Yes, It is possible. The input format class provides methods to insert multiple directories as input to a Hadoop job.

---

17) What is the relation between job and task in Hadoop?

In Hadoop, A job is divided into multiple small parts known as task.

---

18) What is distributed cache in Hadoop?

Distributed cache is a facility provided by MapReduce Framework. It is provided to cache files (text, archives etc.) at the time of execution of the job. The Framework copies the necessary files to the slave node before the execution of any task at that node.

---

19) What commands are used to see all jobs running in the Hadoop cluster and kill a job in LINUX?

Hadoop job - list

Hadoop job - kill jobID

---

20) What is the functionality of JobTracker in Hadoop? How many instances of a JobTracker run on Hadoop cluster?

JobTracker is a giant service which is used to submit and track MapReduce jobs in Hadoop. Only one JobTracker process runs on any Hadoop cluster. JobTracker runs it within its own JVM process.

Functionalities of JobTracker in Hadoop:

• When client application submits jobs to the JobTracker, the JobTracker talks to the NameNode to find the location of the data.
• It locates TaskTracker nodes with available slots for data.
• It assigns the work to the chosen TaskTracker nodes.
• The TaskTracker nodes are responsible to notify the JobTracker when a task fails and then JobTracker decides what to do then. It may resubmit the task on another node or it may mark that task to avoid.

---

21) How JobTracker assign tasks to the TaskTracker?

The TaskTracker periodically sends heartbeat messages to the JobTracker to assure that it is alive. This messages also inform the JobTracker about the number of available slots. This return message updates JobTracker to know about where to schedule task.

SAS Interview questions

Some frequently asked SAS Interview questions

SAS aka Statistical and Analytical Software is one of the commonly used languages for Statistical analysis throughout the world due to its tremendous power for data mining, statistical analysis,

In this post, we will be taking you through the most frequently asked SAS interview questions.
We wont just be providing answers but will be explaining through examples as well.

1.  What is DATA _NULL_ in SAS?

Data _null_ step is one of the very commonly used datasteps in SAS.

It is primarily used to create macro variables for use later in the program.

The reason why "Data _null_" is used is because not to consume unnecessary space by creating a dataset for just creating macro variables.

Example 1:

data _null_;
           length country $15.;

           input country $ num_of_states ;

           put country num_of_states;

           datalines;
                  India 29
                  Brazil 26
                  America 50
;

The following output is written in the SAS log:

India 29

Brazil 26
America 50

Example 2:
data _null_;
   call symput('blogname',"Onespot Anaytics");
run;

The above data _null_ creates the macro variable blogname with the value "Onespot Analytics"  - without creating any dataset.

You can check the value of the macro variable by writing:

%put &blogname.;

The output in the SAS log is:
Onespot Analytics

2.  What are macro variables in SAS?

Macro variables are user-generated variables or symbols that enable you to store text and use them later in the SAS program.

Small or large amounts of texts can be assigned to macro variables, and by simply referencing the macro variable , the text can be referenced.

Some facts about macro variables below:

• The value stored in a macro variable can have a maximum length of 65,534 characters.

• The length of a macro variable is determined by the text assigned to it instead of a specific length declaration.So its length varies with each value it contains.

• Macro variables contain only character data. However, the macro facility has features that enable a variable to be evaluated as a number when it contains character data that can be interpreted as a number. An example to explain this below:

Example:

%macro test(finish);                               /*Create a function test with finish as a parameter to it*/

                %let i=1;                                 /*Create a macro variable "i" with a value of 1*/

                %do %while (&i<&finish);      /*Check whether the "i" is lesser than "finish"*/                                          %put the value of i is &i;

                           %let i= %eval(&i+1);  /*Increment the value of i*/

                %end;

%mend test;

%test(5);

Note: In line 3 of the code above, although i is a macro variable,the "less than" function can be used on it - showing that when a macro variable can be evaluated as a number.

3. What are the different ways by which you can create macro variables in SAS?

There are a total of 4 ways by which macro variables can be created in SAS:

• Using the %let statement

           Example:

           %let i= 1                            /*Creates the macro variable with the value 1*/

• Using call symput

          Example:

          data _null_;
                    call symput('blogname',"Onespot Anaytics");
          run;

          Creates the macro variable "blogname" with the value "Onespot Analytics"

• Using PROC SQL insert into clause

          Example:

          proc sql;
                         select max(age) into :max_age
                         from student;
          quit;
          

          Stores the maximum of variable age from student in the macro variable max_age.

• As a parameter to a macro function

          Example:

          %macro student_details(age,height);

                      %put The age of the student is &age. years and his height is &height. cms.;

          %mend;

          %student_details(17, 178);

          2 macro variables age and height are passed as parameters to the macro function                                 student_details with values 17 and 178 respectively.

4.  In SAS, what can be the minimum length of numeric and character variable respectively?

The minimum length of a numeric variable is 2 bytes whereas the minimum length of a character variable is 1 byte.

5.  How to drop variables/columns from a dataset in SAS?

Drop statement can be used to remove certain variables from a dataset in SAS:

The below example drops the 3 variables age , height and weight from the dataset old_student.

Example:

data new_student(drop= age height weight);
        set old_student;
run;

Another way of writing it would be:

data new_student;
        set old_student;
        drop age height weight;
run;

6.  How to print the first n number of observations from a dataset in SAS?

The obs option can be used to print n observations from a dataset:
Similarly the outobs option can be used to print n observations in proc sql

Example 1:
proc print data=student(obs=10);
run;

Prints the first 10 observations from the dataset student.

proc print data=student(firstobs= 3 obs=10);
run;

Prints the first 10 observations from the dataset student starting from the 3rd row.

Example 2:

proc sql outobs = 10;
         select * from student;
quit;

The OUTOBS= option restricts the number of rows that are displayed, but not the rows that are read.

proc sql outobs = 10 inobs=10;
         select * from student;
quit;

The INOBS= option restricts the number of rows that are read from the student dataset.

7.  What is the difference between SAS PROCs (Procedures) and the SAS DATA STEP?

SAS Procs are inbuilt sub-routines in SAS which serve a specific purpose whereas the data step is a user-created set of steps designed to read in and manipulate data.

Examples of SAS PROCs are:

PROC SORT
PROC SQL
PROC FREQ
PROC CONTENTS
PROC PRINT
PROC MEANS
PROC SUMMARY
PROC TRANSPOSE
PROC TABULATE
PROC REPORT
PROC UNIVARIATE
PROC MULTIVARIATE

8.  What is a Program Data Vector (PDV) in SAS?

A Program Data Vector is a temporary space or buffer space in the SAS memory. Thus the PDV is basically a logical area in the memory.

Let us explain PDV with an example:

data student2;
        set student (keep= name age height);          /*Keep the 3 variables name, age and height*/
run;

The way the dataset student is read into student2 is row by row.
That is , observations are read from student one row at a time.

Each row is first read from the dataset student and stored in the PDV before

The _ERROR_ variable acts like a binary switch whose value is 0 if no errors exist in the DATA step, or 1 if one or more errors exist.

Along with the columns that are read from the dataset , 2 more variables are created in the PDV - _N_ and _ERROR_.

• _N_ denotes the row number currently being read from the dataset.
• _ERROR_ variable acts like a binary switch whose value is 0 if no errors exist in the DATA step, or 1 if one or more errors exist.

The first row from the Student dataset is read as below into the PDV:

This is then output to the Student2 dataset except _N_ and _ERROR_. Student2 now has 1 row.

Next the 2 row is read from Student dataset to the PDV and then to Student2.

9.  What is the difference between INTNX and INTCK functions in SAS?

INTNX and INTCK are interval functions in SAS

The INTNX function can be used to increment/decrement dates by intervals.
                                                       whereasThe INTCK function counts the number of intervals between two date values.

data _null_;
call symput("start_year",intnx( "year" , today() , 0 , "B" ));
call symput("diff_year",intck( "year" , '01JAN2013'D , '10JAN2015'D));
run;

The intnx function above tries to find the starting day of the year of today's date.

The intck function tries to find the difference in the number of years between the 2 dates.

10.  What are _NUMERIC_ and _CHARACTER_ in SAS and what do they do?

• _NUMERIC_ specifies all the numeric variables that are present in the current DATA step.

• _CHARACTER_ specifies all the character variables present in the current DATA step.

• _ALL_ specifies all the variables that are present in the current DATA step.

Example:
Let us assume the dataset student has the following columns:

idnumber name $  age height weight birthplace $

If you want to keep only the numeric variables: 

data student2;
         set student(keep = idnumber - numeric- weight);
run;

11.  What are the different ways in which you can check whether a column in a dataset has unique values or not?

There are several methods to do this:

Lets check if the height variable has duplicate values or not.

• Using proc freq

          proc freq data= student;
                  tables height;
          run;
In the SAS Output window, all values of the variable height along with the frequency is displayed.

• Using proc sort nodupkey & dupout

          proc sort data= student nodupkey dupout= student_dup;
                by height;
          run;
In the student_dup dataset, all duplicate values of the variable height will stored.

• Using first. and last.

          data student_dupl student_unique;                          /*Create the 2 datasets*/

                  set student;

                  if first.height <> last.height then output= student_dupl;

                  else output= student_unique;

          run;

The dataset student_dupl contains the duplicate values of the variable height.

12.  How will you convert a numeric variable into a character variable and a character variable into a numeric variable?

To convert a numeric variable into a character variable, use the put function:
                  char_value = put( numeric_value, $10.)

To convert a character variable into a numeric variable, use the input function:
                  numeric_value= input(char_value, 10.)
           
Another way of converting a character variable into numeric variable is multiplying with 1:
                  numeric_value= char_value * 1;

13.  What are the different data types in SAS?

SAS has only 2 datatypes - numeric and character.

A common question is - isn't date also a datatype.
Actually in SAS, dates are also stored in numeric format - this is explained in one of the later.

14.  What is the difference between NODUP and NODUPKEY options in PROC SORT in SAS?

NODUP searches for duplicate values of all the variables in the dataset while NODUPKEY compares just the BY variables.

proc sort data= student nodupkey;
               by height;
run;

In the above code, records having duplicate values of the variable height are deleted.

proc sort data= student nodup;
                by _all_;
run;

Only those rows in which all the variables have same values will be deleted.

Note: We have used _all_  to sort by all the variables. This is required because the rows having all the variables having same values - must be next to one another. 

This is because SAS can only look one record back.

15.  What does the RUN statement do in SAS?

When the SAS editor encounters the "RUN" statement - it starts compiling the datastep or the proc step.

16.  How does SAS internally represent character and numeric missing values?

Character missing values are represented internally as Blanks or " " and numeric missing values are represented internally as .

17.  Which date is represented by the SAS date value of 1900?

01 January, 1960 is treated as the first date in SAS. That is 01 January, 1960 is treated as SAS date value 1.
Therefore, 1900 SAS date value would be 15 March, 1961.

18.  What are the differences between Proc MEANS and Proc SUMMARY?

Proc SUMMARY and Proc MEANS have the same functionality in SAS and both of the procedures compute descriptive statistics.

However there are 2 differences between the procedures:

1)
Proc MEANS by default prints the output in the Output window whereas Proc SUMMARY does not.
However by using the PRINT option explicitly in Proc SUMMARY - the results can be output to the output window.

2)
The second difference is the behavior of the two procedures when the VAR statement is omitted.

Proc MEANS analyses all numeric variables and produces default statistics for these variables
(N, Mean, Standard Deviation, Minimum and Maximum) whereas Proc SUMMARY simply produces the count of observations if the VAR statement is omitted.

19.  What's the difference between Keep= VAL1 - VAL3 and Keep= VAL1 -- VAL3?

Say the dataset contains the variables VAL1, VAL2, A1, VAL3

• Using the Single Hyphen, only the consecutively numbered variables VAL1, VAL2 and VAL3 will be considered by the Keep statement.

• Using the Double Hyphen, all the the variables between VAL1 and VAL3 will be kept that is, VAL1, VAL2, A1 and VAL3.

20.  Does SAS "Compile" the statements or does it "Interpret"?

SAS compiles the statements.

SAS does not read and execute one statement at a time.

Instead what it does is wait for a RUN or QUIT statement and executes the corresponding DATA step or PROC statement. When the data step is submitted for execution, SAS checks the syntax of the SAS statements and translates them into machine code. During this phase, SAS identifies the type and length of each variable and determines whether a type conversion is necessary for each subsequent reference to a variable.

This behaviour is similar to a Compiler rather than an Interpreter.

21. Why is SAS called as self-documenting?

When a SAS dataset is created, SAS automatically creates the "descriptor" portion as well as the data portion of the dataset.

In the descriptor portion, SAS stores the details like when the dataset was created, the number of observations(rows) in the dataset, the number of variables(columns) in the dataset etc.

Due to this property of SAS to accumulate, record and store the description of the data; SAS is called as self-documenting.

22.  What is the difference between Z= a+b+c+d and
Z= sum(a,b,c,d)?

Using the + operator, if any of the variables contains a missing value, the sum is automatically set to a missing value.

Whereas the SUM function treats the missing value as 0 and computes the sum.

Example:

X= 1 + . + 5 + 7 = .  (as the second value is a missing value)

whereas X= sum(1, . , 5 , 7) = 13 

23.  At compile time, when a SAS dataset is read, what items are created by SAS?

At compile time, the following items are created by SAS:

a) Input Buffer

b) Program Data Vector (PDV)

c) Descriptor information like the time and date when the dataset was created, the number of observation and variables in the dataset.

24.  In SAS Array processing, what does the DIM function do?

DIM is the dimension function in SAS.

It returns the number of elements in the array list.

Example:

In a 1- dimensional array

array big{5} weight sex height state city;
do i=1 to dim(big);
     more SAS statements;
end;

The do loop runs for 5 times as the array big contains 5 elements.

Example:

In a multi-dimensional array, the dim function returns the number of elements in a specified dimension of the multi-dimensional array.

array mult{5,10,2} mult1-mult100;

The above array creates a 3-d array with 5*10*2 elements.

DIM(multi, 1) would return 5.
DIM(multi, 2) would return 10.
DIM(multi, 1) would return 2.