Running Jobs as application user in Cloudera hadoop distribution

When an Hadoop cluster is not enabled with Kerberos authentication, Internally triggered jobs would be running as an 'yarn' user rather than application user.

This blog has option to overcome this limitation. There are 2 settings needs to be done in Cloudera manager (Tested in Cloudera distribution).

1. Unlimit the user in insecure mode [yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users] (YARN->Configuration)

2.  Enable container executor to Linux rather than default.

Application error that you might get:

org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/{some-hadoop-path}":app-user:app-group:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)

at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)

Job started with user 'nobody'

Reason for cause: It means the property "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" is not set to 'false'. Please re-verify.

Diagnostics: Application application_1520446094382_0028 initialization failed (exitCode=255) with output: main : command provided 0

main : run as user is nobody

main : requested yarn user is app-user

Can't create directory /data/3/cloudera/yarn/nm/usercache/app-user/appcache/application_1520446094382_0028 - Permission denied

Did not create any app directories

Failing this attempt. Failing the application.

Job failed with 'app-user' not found
Reason for cause: Some of the nodes may not have app-user created. Create users in all the nodes participating in the cluster.

Diagnostics: Application application_1520465635722_0003 initialization failed (exitCode=255) with output: main : command provided 0

main : run as user is app-user

main : requested yarn user is app-user

User app-user not found

Failing this attempt. Failing the application.

Job failed with permission denied error 
Reason for cause: there might be cache folder created by other/yarn user(s) and app user might not have permission to create folder to write data. To solve you have to delete file or folders under usercache directory, its safe to delete. [Example. 'rm -rf /data/cloudera/yarn/nm/usercache/*' ]

Diagnostics: Application application_1520465635722_0012 initialization failed (exitCode=255) with output: main : command provided 0

main : run as user is app-user

main : requested yarn user is app-user

Can't create directory /data/cloudera/yarn/nm/usercache/app-user/appcache/application_1520465635722_0012 - Permission denied

Did not create any app directories

Failing this attempt. Failing the application.

Viewing maven dependency jars/tree

I had spend hours to identify few additional jars getting included in the war file that causes failure in deployment and error in bringing up the application, especially when container provides those libraries with different version of it.

Luckily there is an option in maven to see the dependency as well dependency of dependencies, which i did not learn so far. The command is,

mvn dependency:tree

and also found that there are more ways to view visual graph.

JUnit fails - with java.lang.ClassFormatError: Absent Code attribute in method that is not native or abstract in class file javax/persistence/Persistence

It is been de facto standard that writing a JUnit test and confirming ORM (Object Relational Mapping) layer perfect before validating service, web layers, whatever may be the methodology are Agile, TDD etc.,

We tend to include API spec jar's and forget to add implementation in maven (pom) dependency file. this would result in java.lang.ClassFormatError: Absent Code attribute in method that is not native or abstract in class. 

This could be resolved by referencing any valid implementation of the specification. For an example you could reference one of the (glassfish, jboss etc) application container implementation jar's should solve this error.

References:

http://www.mkyong.com/hibernate/java-lang-classformaterror-absent-code-attribute-in-method-that-is-not-native-or-abstract-in-class-file/

https://norrisshelton.wordpress.com/2012/04/09/java-lang-classformaterror-absent-code-attribute-in-method-that-is-not-native-or-abstract-in-class-file-javaxpersistencepersistence/

Way to copy file from one machine to another without prompt

Below are the way's to copy files from one machine to another without user interaction.

1) Keybased -  Need to generate a key pairs on the originating (sender) host and have to add public key into remote machine. This is flexible and more convenient way to send files without supplying password.
 

2) sshpass - The tool that we need to install in sender machine and this works wrapper around scp. It requires password to be present in file or some other form so that we can pass it. (http://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/)

3) expect/interact -  This is another way to pass the password via script, it also requires these to be installed in sender machine.

Security Enhanced Linux

Wanted to share one of the Linux folder to the team [windows users ;-)] within the network. Thought it's good to start samba service and followed the step's mentioned in this link.

I was able to start the service but not able to access the folders from windows machine. Looked at the log "/var/log/samba/smbd.log" and noticed below error.

  '/home/someuser' does not exist or permission denied when connecting to [SharedFolder] Error was Permission denied
[2012/08/12 14:03:04, 0] smbd/service.c:make_connection_snum(1003)

I have given all rights and tried all possible debugging, finally found that recent addition of Security Enhanced access control policy denies access to that shared folder.  To find out use the command "getenforce", to disable the enforcement used command "setenforce 0"

It also can be configured in file "/etc/selinux/config"  but requires restart.

Happy Sharing! 

Cobertura

Code coverage is one of the benchmark strategy for any commercial or opensource product.  
Cobertura is one of the best tool to generates code coverage percentage. 
It can be configured both ant and maven build. In maven goal is "cobertura:cobertura" 
refer here other goals.

[INFO] Cobertura Report generation was successful.

You can see in the logs code coverage generation is success but you will 

not see any report or coverage.xml in the work space. To generate 

coverage.xml, need to add below plugin in the pom file.

   <build>
<plugins>
<plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>cobertura-maven-plugin</artifactId>
                <version>2.2</version>
                <configuration>
                    <formats>
                        <format>xml</format>
                    </formats>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>cobertura</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
</plugins>
</build>

This would work better if we have only one module, but if application having multi-module (api, services, web etc)
consolidated report generation having an issue still open at the time of writing.

Below is the error you would get if you have multiple module application.

Skipping Cobertura coverage report as build was not UNSTABLE or better ...

cobertura multi-module coverage

OSGi Logging

Logging is essential for any enterprise application. Looking for best one for our product .. should be slf4j..

http://ekkescorner.wordpress.com/blog-series/osgi-apps/

http://blog.frankel.ch/thoughts-on-java-logging-and-slf4j

http://aappddeevv.blogspot.in/2009/06/logging-in-osgi-and-eclipse-rcp.html