8.5 Stable Release Series 7.2

This is a stable release series of Condor. As usual, only bug fixes (and potentially, ports to new platforms) will be provided in future 7.2.x releases. New features will be added in the 7.3.x development series.

The details of each version are described below.

Version 7.2.4

Release Notes:

• None.

New Features:

• None.

Configuration Variable Additions and Changes:

• None.

Bugs Fixed:

• Fixed a bug in the checkpoint server that caused failure of checkpoint image storage and retrieval if the requesting submission machine was running a 32-bit installation of Condor and the checkpoint server was from a 64 bit installation, or vice versa. The checkpoint image server, both the 32-bit and 64-bit installation, now handles both protocols. It is recommended that any checkpoint server installation which may be used in a flocking situation or other federated joining of pools use the 64-bit binary. This is due to the possibility that there could be a checkpoint image larger than what is representable in 32 bits. A 32-bit checkpoint image server will now notice if this situation occurs and log a message suggesting an upgrade to the 64-bit version.

• Fixed a bug that caused condor_procd to sometimes fail when monitoring processes with environments larger than 1MB.

• Fixed a bug that caused condor_dagman to fail in recovery mode on a DAG in which any nodes had been retried.

• Xen-based virtual machines now have the correct amount of memory. Previously, the amount of memory was too small by a factor of 1024.

• Fixed a bug in the handling of $$(VARIABLE) submit file expressions.

• Fixed a bug in the code related to USE_VISIBLE_DESKTOP that was causing the windows created by the job behave incorrectly.

• Fixed a bug that caused Stork to treat successful file transfers as failed.

• Fixed several bugs in the user log reader in the handling of files of size zero.

• Fixed a problem affecting parallel universe jobs with very short tasks. If any of the parallel tasks exited before the first node started, the entire job was prematurely treated as though it had finished. If the job ClassAd attribute ParallelShutdownPolicy was set to "WAIT_FOR_ALL", the job was prematurely treated as though it had finished if all stated tasks completed before the remaining tasks started.

Known Bugs:

• Occasionally, Condor daemons will, for unknown reasons, bind the command socket to the invalid IP address of 0.0.0.0, resulting in the daemon crashing or otherwise malfunctioning. The command socket address is always logged in the daemon's log, so the condition can be detected by looking for a line like in the daemon's log header with the address of 0.0.0.0 as in:

  5/6 16:34:26 DaemonCore: Command Socket at <0.0.0.0:53795>
  

  If you encounter this problem, please send an e-mail to condor-admin@cs.wisc.edu with any relevant details.

Additions and Changes to the Manual:

• Descriptions and definitions of all commands that may be placed within the submit description file have been moved from the condor_submit manual page to section .

• Added a description of the configuration variable NEGOTIATOR_MATCHLIST_CACHING. See 3.3.17 for the definition.

Version 7.2.3

Release Notes:

• The header files for ClassAds are now included within the release.

New Features:

• Enhanced the Debian 5.0 Condor port on the x86_64 platform to include support for the standard universe.

Configuration Variable Additions and Changes:

• The new integer configuration variable SEC_TCP_SESSION_DEADLINE specifies the number of seconds after which the client should give up its attempt to establish a security session with a daemon that it is connecting to. The default value is 120 seconds.

• The new configuration variables SCHEDD_CLUSTER_INITIAL_VALUE and SCHEDD_CLUSTER_INCREMENT_VALUE are integers that specify the cluster number to use for the first job submission, and the stride used to increment the cluster id upon successive submissions. See 3.3.11 and 3.3.11 for the complete definitions of these variables.

Bugs Fixed:

• Fixed a memory leak in the condor_collector daemon. The growth in memory over time was approximately 10Mbytes per day per 1000 slots. This bug was introduced in Condor version 7.2.0.

• Fixed a problem that caused integrity checking of most UDP packets longer than about 40Kbytes to fail. This bug affected all previous versions of Condor.

• By adding the new configuration variable SEC_TCP_SESSION_DEADLINE, fixed a problem that has existed since Condor version 7.1.2. The problem was that non-blocking read operations in the security handshake had no timeout, and could therefore lead to a socket remaining allocated indefinitely, if the other side of the connection did not respond. When this problem was observed, the following message appeared in the log written by the condor_schedd daemon:

    file descriptor safety level exceeded
  

• Fixed a rarely observed bug in the event log reader code that could cause it to not detect missed events.

• A bug in the Chirp java client has been fixed. The ChirpInputStream's read() method was returning negative values when encountering binary data.

• condor_dagman now rejects negative node retry values.

• condor_dagman no longer generates a rescue DAG if the DAG is aborted, but is considered successful; this is when ABORT-DAG-ON returns the value 0.

• The user log event numbered 27, named "Job submitted to grid resource", is now written for all grid universe jobs. Previously, it was not written for pbs, lsf, nordugrid, or unicore grid types.

• Fixed a bug where a Condor-C job with both remote_<foo> and remote_remote_<foo> attributes would not have a remote_<foo> attribute when submitted to the remote condor_schedd daemon.

• Fixed a bug in condor_configure and condor_install that would leave the configuration variable CONDOR_HOST unset when configuring a central manager without using the -central-manager command-line argument.

• Fixed a bug that could cause the condor_schedd daemon to leak memory and file descriptors when using the EVENT_LOG configuration variable.

• Fixed a bug in the condor_gridmanager that could cause it to not send a clean up signal to the GRAM jobmanager for removed gt2 jobs.

• Fixed a bug that caused parallel jobs to not work when encryption was enabled.

• Fixed a bug in the Windows installer that caused it to fail to start Condor.

Known Bugs:

• None.

Additions and Changes to the Manual:

• None.

Version 7.2.2

Release Notes:

• None.

New Features:

• Added a full port of Condor to Debian 5.0 on the x86 platform.

• Added a clipped port of Condor to Debian 5.0 on the x86_64 platform.

• Added the -DumpRescue command-line flag to condor_dagman and condor_submit_dag. This flag is intended mainly for testing.

• Added support for the -debug option to condor_qedit.

• The Job Router now uses a time slice timer for periodic expression evaluation, similar to the condor_schedd daemon. The evaluation interval is controlled by the configuration variable PERIODIC_EXPR_INTERVAL, and defaults to 60 seconds, the same default value used by the condor_schedd daemon.

• The Job Router now resets the source job, if a failure occurs when updating the condor_schedd daemon for a periodic expression that evaluated to True. The job's periodic expressions should be evaluated again some time in the future with a successful update.

Configuration Variable Additions and Changes:

• The new boolean configuration variable EVENT_LOG_FSYNC provides control of the behavior of Condor when writing events to the event log. Previously, the behavior was as if this parameter were set to False. See 3.3.4 for the complete definition of this variable.

• The new boolean configuration variable EVENT_LOG_LOCKING provides control of the behavior of Condor when writing events to the event log. Previously, the behavior was controlled by ENABLE_USERLOG_LOCKING. See 3.3.4 for the complete definition of this variable.

• The new string configuration variable TRANSFERER specifies the path to the condor_transferer program which is invoked by the condor_replication daemon to perform the actual transfer of the file set by STATE_FILE. This is part of the high availability framework. Prior to Condor 7.2.2, the value of TRANSFERER was hard coded to $(RELEASE_DIR)/sbin/condor_transferer. The use of this hard coded behavior should be considered obsolete behavior, and will be removed in a future version of Condor.

• The PREEMPTION_REQUIREMENTS and the RANK expression in the matchmaker can now reference many more ClassAd attributes than just SubmittorPrio. New attributes allow this expression to take into account resources currently in use, as well as group usage and quota info. New attributes are: SubmitterUserResourcesInUse, RemoteUserResourcesInUse, RemoteGroupResourcesInUse, RemoteGroupQuota, SubmitterGroupResourcesInUse, SubmitterGroupQuota.

• Added JOB_ROUTER_ATTRS_TO_COPY configuration option. This is a comma separated list of attributes that the Job Router should copy from the routed ad to the source ad in addition to internally hard coded attributes that are copied.

• Added JOB_ROUTER_RELEASE_ON_HOLD. configuration option that will control whether the Job Router will reset the source job to an untouched state if it needs to yield the job because the routed job went on hold. The option defaults to resetting the source job.

• The new configuration variables PREEMPTION_REQUIREMENTS_STABLE and PREEMPTION_RANK_STABLE identify for Condor if all attributes in the variables PREEMPTION_REQUIREMENTS and PREEMPTION_RANK will not change within a negotiation interval.

• The new configuration variables OFFLINE_LOG and OFFLINE_EXPIRE_ADS_AFTER specify the location of persistent machine ClassAds for hibernating machines, as well as the lifetime of the persistent ClassAds.

Bugs Fixed:

• Fixed the condor_collector daemon such that hibernating machines never time out.

• Fixed incorrectly set ClassAd attribute values of machines entering a hibernation state. All hibernating machines are unclaimed and idle, they have no load, the CPU is not busy, and the keyboard and console appear as if they had been idle for a long time.

• Fixed a bug where if any idle slot satisfied the HIBERNATE expression, Condor would put the machine into a sleep state irrespective of any active slots.

• Fixed a bug on Windows that made it impossible to use the defined string "S5" for hibernation.

• Fixed a bug in the condor_starter where it would be running as real uid condor after job hooks are invoked which causes issues when accessing files.

• Fixed a bug where some machines would send a final update ad to the condor_collector, invalidating the persistent one that was previously sent (when HIBERNATE evaluates to True). This had the effect of dropping the machine out of the pool once the ad had grown stale.

• Fixed a bug where any two Condor daemons on Windows were able to bind to the same port at the same time.

• Fixed the behavior of the condor_negotiator so that when a Condor-G matchmaking ad matches, the machine's ad will be shuffled to the end for round-robin matching to multiple gatekeepers with the same rank.

• Resolved a bug in which the submit description file command vm_macaddr was improperly parsed, and thus ignored, by condor_submit for vm universe jobs.

• Condor's Windows zip file distribution now includes the new C/C++ runtime libraries.

• Fixed a Windows platform bug for jobs that enable streaming I/O. The bug caused the condor_starter to crash upon invocation of the job.

• Fixed a bug in which an ill-formed network packet could crash a Condor daemon. This would not be seen in normal Condor operation, but sometimes port-scanning software could trigger such a crash.

• Fixed a bug in which condor_q would sometimes exit with the value zero, indicating success, when it could not connect to a condor_schedd daemon. It now exits with an error code.

• Fixed two seemingly small memory leaks in Condor's SOAP interface. A small amount of memory was lost per SOAP transaction. On a high traffic machine, this leak would eventually render the condor_schedd daemon unresponsive.

• Fixed a bug in the parallel universe where periodic expressions involving the JobStatus attribute would not function properly.

• Fixed a bug where Condor daemons could segmentation fault while trying to write a core file to disk in the Unix ports.

• Fixed a bug in which the use of dedicated execute accounts (indicated by use of the configuration variable DEDICATED_EXECUTE_ACCOUNT_REGEXP) did not work properly in PrivSep mode: those with the configuration variable PRIVSEP_ENABLED set to True.

• Fixed an erroneous log message that reported that the hook defined by HOOK_UPDATE_JOB_INFO had run, but would print the $(HOOK_PREPARE_JOB) path. The correct hook ran, so this was only a logging error. The log message is only visible at the D_FULLDEBUG level.

• Fixed a bug that caused condor_dagman to crash if the dagman.out file reached a size of 2 GBytes.

• Fixed a problem affecting the condor_starter when in PrivSep mode. After the user job exited, an error was printed in the condor_starter log file complaining that it failed to chown the sandbox to Condor ownership. This error was not actually harmful, just noisy.

• Fixed a bug in the condor_master that caused it to not have REPLICATION in its default list for DC_DAEMON_LIST. The example configuration file for HAD has been updated to match, as well.

• Fixed the condor_transferer daemon and documentation to consistently use the value of the configuration variable MAX_TRANSFERER_LIFETIME in High Availability code.

• Fixed a bug that caused condor_dagman to crash, if a splice DAG has node categories.

• Changed splice-related condor_dagman debug messages to not be printed at the default verbosity. They are now mostly printed at debug level 4. For definitions of the debug levels, see the condor_dagman manual page at section  9.

• Fixed a bug that caused the condor_replication daemon, as part of the high availability framework, to start the condor_transferer client incorrectly; the end result was that the condor_transferer was unable to authenticate via GSI using host-based certificates.

• Fixed a bug in which the ClassAd attribute RemoteWallClockTime could get too big after a restart of the condor_schedd daemon, for jobs that were running at the time of the restart.

• Fixed a bug that was causing the condor_startd to log the error message

    ioctl(SIOCETHTOOL/GWOL) failed: Operation not permitted (1)
  

  when started as a Personal Condor on Linux. The message is now suppressed in this case. When the message is printed, an additional message is logged informing the user that this error can be ignored, unless hibernation is being used.

• Fixed a bug that was causing the condor_startd to sometimes publish the network adapter's hardware address incorrectly in its ClassAd.

• Fixed a case in which condor_history could get into an infinite loop when searching through a corrupted history file.

• Fixed a bug in the user log reader code that could cause it to get into an inconsistent state after detecting missed events.

• Condor version 7.2.2 and previous releases do not support communication with Condor 7.3.x daemons using the new 7.3.x configuration variables CCB_ADDRESS or PRIVATE_NETWORK_NAME. The version 7.2.2 condor_collector daemon now recognizes when it is receiving ClassAds from such daemons, and it will reject them. In prior versions, Condor would accept the ClassAds, but attempts to use them led to unexpected behavior.

Known Bugs:

• None.

Additions and Changes to the Manual:

• A manual page for condor_power now appears in the manual. condor_power sends a packet to a machine in a low power state, to cause the machine to wake from that state.

• Reorganized the user manual section that describes DAGMan.

• Added a note about the fact that environment values specified with the environment submit description file command override values from the submittor's environment, as imported with getenv = True.

• Added new information to the section on Power Management pertaining to the handling of hibernating machines.

Version 7.2.1

Release Notes:

• This release addresses reported 7.2.0 problems with the Windows distribution.

New Features:

• Condor now has a clipped port to i386 Debian 5.0 (Lenny).

• Added standard universe support for gfortran.

• Added support for standard output and standard error to be greater than 2 Gigabytes.

Configuration Variable Additions and Changes:

• The configuration variable JAVA_MAXHEAP_ARGUMENT now defaults to the value -Xmx1024m. The installation process of Condor resets this value to UNDEFINED in the local configuration file, if the detected JVM is not from Sun Microsystems.

• A new feature has been added to the condor_master that makes it possible to append to the DC_DAEMON_LIST configuration variable, instead of overwriting it. To take advantage of this, place the plus character ('+') as the first character in the DC_DAEMON_LIST definition. For example:

    DAEMON_LIST     = NEW_DAEMON
    DC_DAEMON_LIST  = +NEW_DAEMON
  

• The new configuration variable DAGMAN_COPY_TO_SPOOL controls whether the condor_dagman binary gets copied to the spool directory when a DAG is submitted. See 3.3.26 for details.

• Added -version and -help command line options to condor_submit_dag.

Bugs Fixed:

• Fixed a bug in the condor_collector which could cause it to hang indefinitely while reading network input in rare conditions.

• Fixed a bug in condor_chirp for Windows which was causing it to crash on invocation.

• Fixed a bug in the Windows condor_mail program, which was causing it to become unresponsive when run. If left running, the application also increased its memory consumption.

• Fixed a bug that could cause the condor_schedd to never evaluate periodic expressions.

• Fixed a bug on Unix platforms where condor_configure would provide incorrect defaults for the JAVA_MAXHEAP_ARGUMENT attribute in the installed configuration files. The new current default for Sun Java JVMs is -Xmx1024m.

• Fixed a bug on Unix platforms where condor_configure would imply that using the Unix user root or UID 0 for the -owner option is a good thing. It is not, and would then complain that it could not find user root in the password file.

• Fixed a bug on Unix platforms where condor_configure would emit errors about not being able to execute ldd when installing Condor on the Mac OS X 10.5 platform. condor_configure now correctly detects shared library requirements when installing the Condor binaries on the Mac OS X 10.5 platform.

• Fixed a bug where execute-side daemons started before the condor_credd would fail to match with Windows jobs with run_as_owner set. This condition persisted until the execute-side daemons were either restarted or reconfigured.

• Fixed a problem affecting the Job Router and Condor-C. When jobs spool input files, they enter a temporary hold state, which could trigger actions by a naive periodic remove or release expression. Periodic expressions are no longer evaluated when in this temporary hold state, which has the hold reason "Spooling input data files".

• The example init script condor.boot.generic erroneously claimed that the condor_master would begin sending SIGKILL to child processes after 20 seconds if SIGQUIT (the fast shutdown) failed. The condor_master will actually wait $(SHUTDOWN_FAST_TIMEOUT) seconds, a value that currently defaults to 300 seconds.

• Environment variable names are now properly treated as case-insensitive on Windows. The most common symptom of this bug was the inability to specify a custom PATH environment variable for a job from its submit description file.

• Changed condor_submit -debug to issue a warning when ignoring environment variables. This occurs with getenv = True set in a submit description file.

• Fixed a long-standing memory leak in SOAP interface. This caused the leak of a few hundred bytes of memory for each connection. This could eventually have caused the condor_schedd daemon to crash.

• Fixed Job Router hooks so that their output is properly propagated where appropriate.

• Implemented a fix for the condor_startd that prevents it from crashing if the user specified the configuration variable NUM_SLOTS_TYPE_N, without also specifying SLOT_TYPE_N.

• The sample configuration files now correctly set the default universe to vanilla. This default has been true since 7.2.0, but was not reflected in the sample configuration files.

• Fixed a bug that incorrectly set the value of the job ClassAd attribute RequestMemory to be 1024 times its correct size due to a mismatch in units; the attribute RequestMemory is given in Mbytes, while the attribute ImageSize is given in Kbytes.

• Fixed a memory leak in condor_dagman that leaked a small amount of memory for each job submitted.

• Fixed a bug that was causing the network mask to be advertised as a Condor sinful string, rather than a dotted-quad.

• Fixed a handle leak in the condor_procd on Windows.

Known Bugs:

• None.

Additions and Changes to the Manual:

• Added a FAQ entry for Windows describing how machines with miss-configured performance counters may cause the condor_procd to crash.

• Added a manual page for the command condor_router_history.

Version 7.2.0

Release Notes:

• A bug in some older Xen kernels can result in Condor errors due to a broken assumption in the condor_procd daemon. See the FAQ entry at section  7.7 for details.

• A problem has been discovered when using snapshot disks with vm universe VMware jobs, if the path that the condor_vm-gahp uses to refer to the virtual machine's VMX file contains a symbolic link. See the FAQ entry at section  7.3 for details.

• The name of the Amazon EC2 GAHP binary has changed from amazon-gahp to amazon_gahp. This makes it consistent with the naming of other Condor binaries.

New Features:

• The default universe for jobs is now vanilla, instead of standard. The default can be changed using the configuration variable DEFAULT_UNIVERSE .

• VMware vm universe jobs now have any BIOS settings saved in an nvram file in the vmware_dir given in the job's submit file transferred to the execute machine, so that they apply to the job's execution.

• Daemons that become unresponsive are now killed using the SIGABRT signal, which causes a core file to be dropped. Setting the configuration variable NOT_RESPONDING_WANT_CORE to False will revert to the previous behavior that used the SIGKILL signal.

• The condor_job_router and the condor_q command with the -better-analyze option now support more ClassAd functions than they previously did. They now support all ClassAd functions, except for those with names beginning with the string stringList.

• condor_status given the options -submitters -xml no longer emits a single blank line when there are no submitters, instead it prints valid XML output with an empty body.

Configuration Variable Additions and Changes:

• The HAD configuration variable NEGOTIATOR_STATE_FILE has changed its name to STATE_FILE.

Bugs Fixed:

• Security Item: A flaw was found and fixed that could allow an unauthenticated user to cause Condor daemons to shut down, and could allow running jobs to be removed from the queue.

• Fixed a bug that caused condor_dagman to stay in the Condor queue, if condor_dagman was accidentally submitted with an empty DAG input file.

• condor_submit_dag now generates a .condor.sub file with the submit description file command copy_to_spool set to True, to ease version upgrades while large DAGs are running.

• Fixed a problem in the condor_startd when using STARTD_SLOT_EXPRS for attributes that are sometimes present and sometimes absent from the machine ClassAd. This is most typical of attributes that enter the machine ClassAd from the job, via STARTD_JOB_EXPRS. When the attribute went away from slot X (for example, because the job on slot X finished), the corresponding SlotX_<AttributeName> attribute was not reliably removed from all of the other slots.

• Removed some redundant information from the condor_startd advertisements to the condor_collector, from within the private ClassAd that is not user-visible. This fix reduces UDP traffic and memory usage generated by the condor_startd by about 20% in the condor_collector and condor_negotiator daemons.

• Fixed the condor_master daemon to correctly preserve all command-line arguments when restarting itself. In some cases, not preserving argv[0] confused external utilities that monitor the condor_master process by looking at the output of ps or similar programs. Also, not preserving -pid and -runfor could cause unexpected behavior.

• Fixed a bug that exhibited itself when the configuration variable NEGOTIATOR_CONSIDER_PREEMPTION was set to False, in which jobs would not be matched to slots in the backfill state. Corrected, slots doing backfill are included in the matchmaking process.

• The condor_job_router did not work while managing jobs from multiple users when read access to the condor_schedd required authentication. The condor_job_router was also not able to use authentication methods other than FS. Now it can use any authentication method, as long as the resulting identity is listed in the configuration variable QUEUE_SUPER_USERS or the condor_job_router and condor_schedd are running as a Personal Condor in non-root mode.

• Fixed a bug in the condor_schedd daemon that could cause it to write an incorrect Unique ID to the event log's header.

• Fixed a bug in the user log reader API that could cause it to incorrectly return a ULOG_NO_EVENT in rare cases.

• Fixed a bug in the user log reader API that could cause it to crash if the application attempted to re-initialize the ReadUserLog object. The code now detects this condition, and returns an error when the application attempts to re-initialization an already initialized ReadUserLog object.

• Fixed a bug that limited the size of stdin, stdout, and stderr files in the vanilla universe to 2GBytes.

• Fixed a bug that could cause the condor_starter to EXCEPT upon completion or eviction of a vm universe job. The error message that appeared in the StarterLog file was

    Write_Pipe: invalid pipe end
  

• When a held job is removed, the values of the attributes HoldReason, HoldReasonCode and HoldReasonSubCode are moved to LastHoldReason, LastHoldReasonCode and LastHoldReasonSubCode. Before, a hold reason could be lost if a removed job was subsequently held.

• The executable attribute for amazon grid universe jobs no longer needs to be a valid file path.

• Improved error reporting when a Xen or VMware command fails in the vm universe.

• For vm universe jobs, virtual floppy disks are no longer disabled.

• Fixed a bug introduced in Condor 7.1.4 that caused Condor to ignore the virtual machine status reported by Xen in the vm universe.

• Fixed a 20-second delay in the start up of the condor_c-gahp and the condor_vm-gahp.

• Fixed a bug which caused the net mask to be published into the machine ClassAd incorrectly.

• Fixed a bug introduced in Condor 7.1.4 which could cause any Condor daemon to crash if the level of debugging output D_ALL is enabled when a condor_reconfig command is issued.

• Fixed a bug introduced in Condor 7.1.4 which caused standard universe jobs to fail to start up, if security authentication, but not encryption was enabled between the submit side and the execute side.

• Fixed a bug with streaming stdin, stdout, and stderr when using glexec.

Known Bugs:

• None.

Additions and Changes to the Manual:

• Initial documentation for dynamic provisioning is available in section  3.13.9.

• Documentation for Kerberos authentication (see section  3.6.3) and associated configuration variables has been updated.