When an Hadoop cluster is not enabled with Kerberos authentication, Internally triggered jobs would be running as an 'yarn' user rather than application user.
This blog has option to overcome this limitation. There are 2 settings needs to be done in Cloudera manager (Tested in Cloudera distribution).
1. Unlimit the user in insecure mode [yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users] (YARN->Configuration)
2. Enable container executor to Linux rather than default.
Application error that you might get:
Job started with user 'nobody'
Reason for cause: It means the property "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" is not set to 'false'. Please re-verify.
Job failed with 'app-user' not found
Reason for cause: Some of the nodes may not have app-user created. Create users in all the nodes participating in the cluster.
Job failed with permission denied error
Reason for cause: there might be cache folder created by other/yarn user(s) and app user might not have permission to create folder to write data. To solve you have to delete file or folders under usercache directory, its safe to delete. [Example. 'rm -rf /data/cloudera/yarn/nm/usercache/*' ]
This blog has option to overcome this limitation. There are 2 settings needs to be done in Cloudera manager (Tested in Cloudera distribution).
1. Unlimit the user in insecure mode [yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users] (YARN->Configuration)
2. Enable container executor to Linux rather than default.
Application error that you might get:
org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/{some-hadoop-path}":app-user:app-group:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:262)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:281)
Job started with user 'nobody'
Reason for cause: It means the property "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" is not set to 'false'. Please re-verify.
Diagnostics: Application application_1520446094382_0028 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is nobody
main : requested yarn user is app-user
Can't create directory /data/3/cloudera/yarn/nm/usercache/app-user/appcache/application_1520446094382_0028 - Permission denied
Did not create any app directories
Failing this attempt. Failing the application.
Job failed with 'app-user' not found
Reason for cause: Some of the nodes may not have app-user created. Create users in all the nodes participating in the cluster.
Diagnostics: Application application_1520465635722_0003 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is app-user
main : requested yarn user is app-user
User app-user not found
Failing this attempt. Failing the application.
Job failed with permission denied error
Reason for cause: there might be cache folder created by other/yarn user(s) and app user might not have permission to create folder to write data. To solve you have to delete file or folders under usercache directory, its safe to delete. [Example. 'rm -rf /data/cloudera/yarn/nm/usercache/*' ]
Diagnostics: Application application_1520465635722_0012 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is app-user
main : requested yarn user is app-user
Can't create directory /data/cloudera/yarn/nm/usercache/app-user/appcache/application_1520465635722_0012 - Permission denied
Did not create any app directories
Failing this attempt. Failing the application.
No comments:
Post a Comment