Issue15898
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2012-09-10 00:58 by amoffat, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (12) | |||
---|---|---|---|
msg170147 - (view) | Author: Andrew Moffat (amoffat) | Date: 2012-09-10 00:58 | |
I'm getting some kind of race condition on OSX when trying to read the output from a pseudoterminal of a forked process. This race conditional only occurs if I run tty.tcsetattr on the master side of the pty, regardless of if I actually change the mode or not. Try running the following code on OSX multiple times. Sometimes you'll see "testing", sometimes you won't, sometimes it hangs. If you comment out "tty.tcsetattr", you will consistently see "testing". import os import pty import resource import signal import tty master, slave = pty.openpty() pid = os.fork() # BUG IS HERE # we're not making any changes to the tty mode, but # the mere act of setting a mode causes the output to only # show up sometimes. # # comment out "tty.tcsetattr" and the bug goes away mode = tty.tcgetattr(master) tty.tcsetattr(master, tty.TCSANOW, mode) # child process if pid == 0: os.setsid() os.close(master) os.dup2(slave, 0) os.dup2(slave, 1) os.dup2(slave, 2) max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0] os.closerange(3, max_fd) # make controlling terminal. taken from pty.fork tmp_fd = os.open(os.ttyname(1), os.O_RDWR) os.close(tmp_fd) os.write(1, "testing".encode()) os._exit(255) # parent process else: os.close(slave) try: print(os.read(master, 1024)) finally: os.kill(pid, signal.SIGKILL) |
|||
msg170150 - (view) | Author: Ned Deily (ned.deily) * | Date: 2012-09-10 02:25 | |
I have not been able to reproduce the behavior you report using various Pythons on OS X 10.8 or on OS X 10.5. Can you give more specifics on the environment that fails for you? I certainly don't claim to have any experience with these tty/pty modules or with the underlying system calls and I'm not sure what you are trying to do here but isn't it a bit risky to be calling tty.tcsetattr() from both the parent and child processes simultaneously, since the call is after the fork? tcsetattr is dealing with a job-wide data structure and, under the covers, the tcsetattr() function currently reads and writes various field non-atomically. What happens if you move the tcgetattr and tcsetattr to before the os.fork call or into the parent-only code after the "else"? http://hg.python.org/cpython/file/default/Modules/termios.c |
|||
msg170153 - (view) | Author: Andrew Moffat (amoffat) | Date: 2012-09-10 02:42 | |
If I move tcsetattr into the parent, the race condition still exists. If I move it into the child, before the write(), it works. My environment is OSX 10.7.2 inside of Virtualbox (maybe this is the problem?) I've shuffled some code around, and found case that fails consistently in python 3, but succeeds consistently on python 2.7. It doesn't use tcsetattr at all, which is making me realize the problem may be deeper than I originally thought. Are you able to confirm this, Ned? import os import pty import resource import signal import tty import time master, slave = pty.openpty() pid = os.fork() # child process if pid == 0: os.setsid() os.close(master) os.dup2(slave, 0) os.dup2(slave, 1) os.dup2(slave, 2) max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0] os.closerange(3, max_fd) # make controlling terminal. taken from pty.fork tmp_fd = os.open(os.ttyname(1), os.O_RDWR) os.close(tmp_fd) os.write(1, "testing".encode()) os._exit(255) # parent process else: time.sleep(0.5) os.close(slave) try: print(os.read(master, 1024)) finally: os.kill(pid, signal.SIGKILL) |
|||
msg170161 - (view) | Author: Ned Deily (ned.deily) * | Date: 2012-09-10 06:55 | |
Sorry, all I can tell you is that, for me, your test produces an empty string as output with either Python 2.7 or 3.3 and, if I remove the time.sleep() call or remove the os.close(slave), "testing" or b"testing" is output. I'm not sure what you believe the problem to be or what behavior you expect; you're covering a lot of ground here and much of it is very close to the OS. You are going to need to be a lot more specific than "OSX TTY bug". You should describe exactly what failing behavior you observe with what component of Python, 2. how to reproduce it, and 3. exactly what behavior you expect of that component, and 4. if possible, the results of testing on another supported platform. Otherwise, it will be very difficult to help. |
|||
msg170164 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2012-09-10 07:42 | |
If you want to analyse this further, please provide "dtruss -f" output of the script. Most likely, this is a glitch in the operating system, but not a bug in Python. Unless you can provide more details, or a patch, we are likely to close this issue as "works for me". |
|||
msg170216 - (view) | Author: Andrew Moffat (amoffat) | Date: 2012-09-10 18:47 | |
@Ned, Ok, give me a little time to work up those suggestions. On my machine, the previous script prints "testing" consistently on 2.7, "" consistently on 3.2. I guess this is the behavior that is unexpected...it should print the same thing for both versions. But if you are unable to confirm it, that's no good. Let me work on this a little more before you guys close this. @Martin, Thanks for the tip. |
|||
msg170261 - (view) | Author: Andrew Moffat (amoffat) | Date: 2012-09-11 04:19 | |
Below I attached a script that reproduces the behavior of an assertion error on a specific fork condition for a process with a pty. Here are the results Xubuntu x64: Python 2.7, parent_close_slave_before_child_write = False: WORKS Python 2.7, parent_close_slave_before_child_write = True: WORKS Python 3.2, parent_close_slave_before_child_write = False: WORKS Python 3.2, parent_close_slave_before_child_write = True: WORKS Mac OSX 10.7.2: Python 2.7, parent_close_slave_before_child_write = False: WORKS Python 2.7, parent_close_slave_before_child_write = True: WORKS Python 3.2, parent_close_slave_before_child_write = False: >> FAILS << Python 3.2, parent_close_slave_before_child_write = True: WORKS Here's dtruss -f for python 2.7 and 3.0 on Mac OSX: 2.7, parent_close_slave_before_child_write = False (WORKS) PID/THRD SYSCALL(args) = return 356/0xc4a: open_nocancel("/dev/ptmx\0", 0x20002, 0x0) = 3 0 356/0xc4a: ioctl(0x3, 0x20007454, 0xFFFFFFFF) = 0 0 356/0xc4a: ioctl(0x3, 0x20007452, 0x0) = 0 0 356/0xc4a: ioctl(0x3, 0x40807453, 0x10041AD20) = 0 0 356/0xc4a: stat64("/dev/ttys003\0", 0x7FFF5FBFF030, 0x0) = 0 0 356/0xc4a: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 4 0 356/0xc4a: fork() = 362 0 362/0xc61: fork() = 0 0 362/0xc61: thread_selfid(0x100420BE0, 0x3, 0x1) = 3169 0 362/0xc61: getpid(0x100420BE0, 0x3, 0x0) = 362 0 362/0xc61: select(0x0, 0x0, 0x0, 0x0, 0x7FFF5FBFF3C0) = 0 0 362/0xc61: setsid(0x0, 0x0, 0x1001A7B40) = 362 0 362/0xc61: close(0x3) = 0 0 362/0xc61: dup2(0x4, 0x0, 0x0) = 0 0 362/0xc61: dup2(0x4, 0x1, 0x0) = 1 0 362/0xc61: dup2(0x4, 0x2, 0x0) = 2 0 362/0xc61: getrlimit(0x1008, 0x7FFF5FBFF3D0, 0x0) = 0 0 ... close range ... 362/0xc61: ioctl(0x1, 0x40487413, 0x7FFF5FBFF360) = 0 0 362/0xc61: fstat64(0x1, 0x7FFF5FBFF2D0, 0x0) = 0 0 362/0xc61: open_nocancel("/dev/\0", 0x100004, 0x0) = 3 0 362/0xc61: fcntl_nocancel(0x3, 0x2, 0x1) = 0 0 362/0xc61: fstatfs64(0x3, 0x7FFF5FBFE7B0, 0x0) = 0 0 362/0xc61: getdirentries64(0x3, 0x1010AB600, 0x1000) = 3080 0 ... lots of lstat64 ... 362/0xc61: close_nocancel(0x3) = 0 0 362/0xc61: open("/dev/ttys003\0", 0x2, 0x1FF) = 3 0 362/0xc61: close(0x3) = 0 0 362/0xc61: write(0x1, "testing\0", 0x7) = 7 0 362/0xc61: close(0x1) = 0 0 356/0xc4a: select(0x0, 0x0, 0x0, 0x0, 0x7FFF5FBFF3C0) = 0 0 356/0xc4a: close(0x4) = 0 0 356/0xc4a: read(0x3, "testing\0", 0x400) = 7 0 356/0xc4a: write_nocancel(0x1, "'testing'\n\0", 0xA) = 10 0 356/0xc4a: wait4(0x16A, 0x7FFF5FBFF3D4, 0x0) = 362 0 356/0xc4a: sigaction(0x2, 0x7FFF5FBFF800, 0x7FFF5FBFF830) = 0 0 3.0, parent_close_slave_before_child_write = False (FAILS) PID/THRD SYSCALL(args) = return 385/0xe53: open_nocancel("/dev/ptmx\0", 0x20002, 0xD15EB8) = 3 0 385/0xe53: ioctl(0x3, 0x20007454, 0xBFFFF1BC) = 0 0 385/0xe53: ioctl(0x3, 0x20007452, 0xBFFFF1BC) = 0 0 385/0xe53: ioctl(0x3, 0x40807453, 0x64F220) = 0 0 385/0xe53: stat64("/dev/ttys003\0", 0xBFFFF164, 0x64F220) = 0 0 385/0xe53: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 4 0 385/0xe53: fork() = 389 0 389/0xe5c: fork() = 0 0 389/0xe5c: thread_selfid(0x0, 0x0, 0x0) = 3676 0 389/0xe5c: getpid(0x0, 0x0, 0x0) = 389 0 389/0xe5c: select_nocancel(0x0, 0x0, 0x0) = 0 0 389/0xe5c: setsid(0x0, 0x0, 0x0) = 389 0 389/0xe5c: close_nocancel(0x3) = 0 0 389/0xe5c: dup2(0x4, 0x0, 0x0) = 0 0 389/0xe5c: dup2(0x4, 0x1, 0x0) = 1 0 389/0xe5c: dup2(0x4, 0x2, 0x0) = 2 0 389/0xe5c: getrlimit(0x8, 0xBFFFF46C, 0x0) = 0 0 ... close range ... 389/0xe5c: ioctl(0x1, 0x402C7413, 0xBFFFF410) = 0 0 389/0xe5c: fstat64(0x1, 0xBFFFF3A4, 0xBFFFF410) = 0 0 389/0xe5c: open_nocancel("/dev/\0", 0x100004, 0xBFFFE968) = 3 0 389/0xe5c: fcntl_nocancel(0x3, 0x2, 0x1) = 0 0 389/0xe5c: fstatfs64(0x3, 0xBFFFE8D8, 0x1) = 0 0 389/0xe5c: getdirentries64(0x3, 0x10DFC00, 0x1000) = 3080 0 ... lots of lstat64 ... 389/0xe5c: close_nocancel(0x3) = 0 0 389/0xe5c: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 3 0 389/0xe5c: close_nocancel(0x3) = 0 0 389/0xe5c: write_nocancel(0x1, "testing\0", 0x7) = 7 0 389/0xe5c: close_nocancel(0x1) = 0 0 385/0xe53: select_nocancel(0x0, 0x0, 0x0) = 0 0 385/0xe53: close_nocancel(0x4) = 0 0 385/0xe53: read_nocancel(0x3, "\0", 0x400) = 0 0 385/0xe53: write_nocancel(0x1, "b''\n\0", 0x4) = 4 0 385/0xe53: wait4_nocancel(0x185, 0xBFFFF474, 0x0) = 389 0 385/0xe53: sigaction(0x2, 0xBFFFF800, 0xBFFFF838) = 0 0 385/0xe53: madvise(0xDA0000, 0x20000, 0x9) = 0 0 385/0xe53: madvise(0xDC0000, 0x20000, 0x9) = 0 0 385/0xe53: madvise(0xD80000, 0x20000, 0x9) = 0 0 2.7, parent_close_slave_before_child_write = True (WORkS) PID/THRD SYSCALL(args) = return 363/0xcab: open_nocancel("/dev/ptmx\0", 0x20002, 0x0) = 3 0 363/0xcab: ioctl(0x3, 0x20007454, 0xFFFFFFFF) = 0 0 363/0xcab: ioctl(0x3, 0x20007452, 0x0) = 0 0 363/0xcab: ioctl(0x3, 0x40807453, 0x10041AD20) = 0 0 363/0xcab: stat64("/dev/ttys003\0", 0x7FFF5FBFF030, 0x0) = 0 0 363/0xcab: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 4 0 363/0xcab: fork() = 368 0 368/0xcb5: fork() = 0 0 368/0xcb5: thread_selfid(0x100420BE0, 0x3, 0x1) = 3253 0 368/0xcb5: getpid(0x100420BE0, 0x3, 0x0) = 368 0 363/0xcab: select(0x0, 0x0, 0x0, 0x0, 0x7FFF5FBFF3C0) = 0 0 363/0xcab: close(0x4) = 0 0 368/0xcb5: select(0x0, 0x0, 0x0, 0x0, 0x7FFF5FBFF3C0) = 0 0 368/0xcb5: setsid(0x0, 0x0, 0x1001A7B40) = 368 0 368/0xcb5: close(0x3) = 0 0 368/0xcb5: dup2(0x4, 0x0, 0x0) = 0 0 368/0xcb5: dup2(0x4, 0x1, 0x0) = 1 0 368/0xcb5: dup2(0x4, 0x2, 0x0) = 2 0 368/0xcb5: getrlimit(0x1008, 0x7FFF5FBFF3D0, 0x0) = 0 0 ... close range ... 368/0xcb5: ioctl(0x1, 0x40487413, 0x7FFF5FBFF360) = 0 0 368/0xcb5: fstat64(0x1, 0x7FFF5FBFF2D0, 0x0) = 0 0 368/0xcb5: open_nocancel("/dev/\0", 0x100004, 0x0) = 3 0 368/0xcb5: fcntl_nocancel(0x3, 0x2, 0x1) = 0 0 368/0xcb5: fstatfs64(0x3, 0x7FFF5FBFE7B0, 0x0) = 0 0 368/0xcb5: getdirentries64(0x3, 0x1010AB600, 0x1000) = 3080 0 ... lots of lstat64 ... 368/0xcb5: close_nocancel(0x3) = 0 0 368/0xcb5: open("/dev/ttys003\0", 0x2, 0x1FF) = 3 0 368/0xcb5: close(0x3) = 0 0 368/0xcb5: write(0x1, "testing\0", 0x7) = 7 0 368/0xcb5: close(0x1) = 0 0 363/0xcab: read(0x3, "testing\0", 0x400) = 7 0 363/0xcab: write_nocancel(0x1, "'testing'\n\0", 0xA) = 10 0 363/0xcab: wait4(0x170, 0x7FFF5FBFF3D4, 0x0) = 368 0 363/0xcab: sigaction(0x2, 0x7FFF5FBFF800, 0x7FFF5FBFF830) = 0 0 3.0, parent_close_slave_before_child_write = True (WORkS) PID/THRD SYSCALL(args) = return 393/0xed3: open_nocancel("/dev/ptmx\0", 0x20002, 0xD15EB8) = 3 0 393/0xed3: ioctl(0x3, 0x20007454, 0xBFFFF1BC) = 0 0 393/0xed3: ioctl(0x3, 0x20007452, 0xBFFFF1BC) = 0 0 393/0xed3: ioctl(0x3, 0x40807453, 0x64F220) = 0 0 393/0xed3: stat64("/dev/ttys003\0", 0xBFFFF164, 0x64F220) = 0 0 393/0xed3: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 4 0 393/0xed3: fork() = 399 0 399/0xee3: fork() = 0 0 399/0xee3: thread_selfid(0x0, 0x0, 0x0) = 3811 0 399/0xee3: getpid(0x0, 0x0, 0x0) = 399 0 393/0xed3: select_nocancel(0x0, 0x0, 0x0) = 0 0 393/0xed3: close_nocancel(0x4) = 0 0 399/0xee3: select_nocancel(0x0, 0x0, 0x0) = 0 0 399/0xee3: setsid(0x0, 0x0, 0x0) = 399 0 399/0xee3: close_nocancel(0x3) = 0 0 399/0xee3: dup2(0x4, 0x0, 0x0) = 0 0 399/0xee3: dup2(0x4, 0x1, 0x0) = 1 0 399/0xee3: dup2(0x4, 0x2, 0x0) = 2 0 399/0xee3: getrlimit(0x8, 0xBFFFF46C, 0x0) = 0 0 ... close range ... 399/0xee3: ioctl(0x1, 0x402C7413, 0xBFFFF410) = 0 0 399/0xee3: fstat64(0x1, 0xBFFFF3A4, 0xBFFFF410) = 0 0 399/0xee3: open_nocancel("/dev/\0", 0x100004, 0xBFFFE968) = 3 0 399/0xee3: fcntl_nocancel(0x3, 0x2, 0x1) = 0 0 399/0xee3: fstatfs64(0x3, 0xBFFFE8D8, 0x1) = 0 0 399/0xee3: getdirentries64(0x3, 0x10DFC00, 0x1000) = 3080 0 ... lots of lstat64 ... 399/0xee3: close_nocancel(0x3) = 0 0 399/0xee3: open_nocancel("/dev/ttys003\0", 0x20002, 0x0) = 3 0 399/0xee3: close_nocancel(0x3) = 0 0 399/0xee3: write_nocancel(0x1, "testing\0", 0x7) = 7 0 399/0xee3: close_nocancel(0x1) = 0 0 393/0xed3: read_nocancel(0x3, "testing\0", 0x400) = 7 0 393/0xed3: write_nocancel(0x1, "b'testing'\n\0", 0xB) = 11 0 393/0xed3: wait4_nocancel(0x18F, 0xBFFFF474, 0x0) = 399 0 393/0xed3: sigaction(0x2, 0xBFFFF800, 0xBFFFF838) = 0 0 393/0xed3: madvise(0xDA0000, 0x20000, 0x9) = 0 0 393/0xed3: madvise(0xDC0000, 0x20000, 0x9) = 0 0 393/0xed3: madvise(0xD80000, 0x20000, 0x9) = 0 0 ----------- import os import pty import resource import signal import tty import time import sys PY3 = sys.version_info.major == 3 if PY3: raw_input = input # change this to False for failure on py3 parent_close_slave_before_child_write = True print("pid: %d" % os.getpid()) raw_input("hit enter when 'dtruss -f' is ready> ") send_from_child = "testing123".encode() master, slave = pty.openpty() pid = os.fork() # child process if pid == 0: if parent_close_slave_before_child_write: time.sleep(2) else: time.sleep(1) os.setsid() os.close(master) os.dup2(slave, 0) os.dup2(slave, 1) os.dup2(slave, 2) max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0] os.closerange(3, max_fd) # make controlling terminal. taken from pty.fork tmp_fd = os.open(os.ttyname(1), os.O_RDWR) os.close(tmp_fd) os.write(1, send_from_child) os.close(1) os._exit(255) # parent process else: if parent_close_slave_before_child_write: time.sleep(1) else: time.sleep(2) os.close(slave) actual_output = os.read(master, 1024) os.waitpid(pid, 0) assert(actual_output == send_from_child) print("correct output") |
|||
msg170264 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2012-09-11 05:19 | |
To me, this still looks like an OS bug. The difference between 2.7 and 3.2 seems to be that 3.2 will call the _nocancel syscalls, which I guess results from using a different version of the C library, or OSX deployment target, or something. However, the system calls are almost identical, see http://fxr.watson.org/fxr/source/bsd/kern/sys_generic.c?v=xnu-1699.24.8#L448 (i.e. the "cancel" version calls __pthread_testcancel, which may result in EINTR, but not in this trace). So ISTM that in the failing case, the child successfully writes to the file descriptor (7 bytes written), yet the parent reads only 0 bytes out of the pty, which makes it look like the kernel discards the data. The trace isn't exactly from the script you provided, right? Because the script sends the string "testing123", whereas the trace shows "testing". |
|||
msg170267 - (view) | Author: Andrew Moffat (amoffat) | Date: 2012-09-11 07:11 | |
Sorry about that, I refactored out the string at the last minute and typed "testing123" instead of the original "testing" from the trace. I'm not sure I follow the problem exactly. So you're saying that the failing version uses _cancel sys calls, and that __pthread_testcancel may result in EINTR. But that this is not happening in the trace, and data is being written successfully. I guess I'm wondering where the written bytes went and if there's any way to retrieve them? |
|||
msg170360 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2012-09-12 08:41 | |
Am 11.09.2012 09:11, schrieb Andrew Moffat: > I'm not sure I follow the problem exactly. So you're saying that the > failing version uses _cancel sys calls, and that __pthread_testcancel > may result in EINTR. But that this is not happening in the trace, > and data is being written successfully. I guess I'm wondering where > the written bytes went and if there's any way to retrieve them? That's exactly the question, and why I say that is must be an OS bug. According to the trace, the OS accepted the data, so if they got lost, it's their "fault". Unfortunately, the dtruss doesn't decode the ioctl arguments, so it's not easy to tell whether they are right. You could try running it with -a to see whether this gives more information. Please don't paste the traces, though, but attach them (the editing is surely appreciated, though). |
|||
msg347914 - (view) | Author: Ronald Oussoren (ronaldoussoren) * | Date: 2019-07-14 13:56 | |
This is IMHO not a bug in Python, the problem can been seen when you rewrite the code in msg170261 in C, see the code below. An observation with the C code below: Both moving ``close(slave)`` to above the sleep or removing that close entirely fixes the problem for me. Likewise with adding ``usleep(700000);`` to the child before exiting. It is unclear to me who's at fault here, this could be a bug in the macOS kernel but could also be a bug in this code. It looks like the output is lost when ``close(slave)`` happens after the child proces has exited. BTW. All testing was done on macOS 10.14.5, with python 3.8. Anyway: I propose closing this issue because this is not a bug in CPython. /* The C code used to test system behaviour */ #include <signal.h> #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <util.h> int main(void) { int master, slave; int r; pid_t pid; r = openpty(&master, &slave, NULL, NULL, NULL); if (r == -1) { perror("openpty"); exit(1); } pid = fork(); if (pid == 0) { /* child */ setsid(); close(master); dup2(slave, 0); dup2(slave, 1); dup2(slave, 2); close(slave); write(1, "testing", 7); _exit(255); } else { /* parent */ char buf[1024]; usleep(500000); /* 0.5 */ close(slave); r = read(master, buf, 1024); if (r == -1) { perror("read"); exit(1); } printf("%d\n", r); write(1, buf, r); kill(pid, SIGKILL); } } |
|||
msg382869 - (view) | Author: Ronald Oussoren (ronaldoussoren) * | Date: 2020-12-11 14:51 | |
Closing this issue, because this is a platform bug. The reproducer in msg170147 works reliably for me on macOS 10.15 with Python 3.9, the system bug may have been fixed in the meantime. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:57:35 | admin | set | github: 60102 |
2020-12-11 14:51:49 | ronaldoussoren | set | status: pending -> closed messages: + msg382869 components: + macOS resolution: not a bug -> third party stage: resolved |
2019-07-14 13:56:29 | ronaldoussoren | set | status: open -> pending resolution: not a bug messages: + msg347914 versions: + Python 3.7, Python 3.8, Python 3.9, - Python 2.6, Python 3.1, Python 3.2 |
2012-09-14 16:21:27 | ronaldoussoren | set | nosy:
+ ronaldoussoren |
2012-09-12 08:41:59 | loewis | set | messages: + msg170360 |
2012-09-11 07:11:29 | amoffat | set | messages: + msg170267 |
2012-09-11 05:19:18 | loewis | set | messages: + msg170264 |
2012-09-11 04:19:30 | amoffat | set | messages: + msg170261 |
2012-09-10 18:47:54 | amoffat | set | messages: + msg170216 |
2012-09-10 07:42:21 | loewis | set | nosy:
+ loewis messages: + msg170164 |
2012-09-10 06:55:04 | ned.deily | set | messages: + msg170161 |
2012-09-10 02:42:20 | amoffat | set | messages: + msg170153 |
2012-09-10 02:25:44 | ned.deily | set | nosy:
+ ned.deily messages: + msg170150 |
2012-09-10 00:58:13 | amoffat | create |