So, all those surveys...

[Raam Sho'am]

Hmm. In order:

1. For starters, I would provide a definition of bullying and ask people to judge based on that. That wouldn't be enough, but it's a start. However, I was not the one who claimed that these numbers were supposed to be objective.

2. You most assuredly cannot tell me that I am wrong. Just because you got a decent number of responses from people who did not feel bullied does not change the fact that those who DID feel bullied were far more likely to respond. There's a reason professional pollsters try their best to achieve random samples.

3 and 4. Then adjust the numbers accordingly. I want to know the percentages based on the number of people who actually answered a question. If someone does not participate in chat, it might be a bit difficult for that person to see bullying there.

5. First of all, I'll thank you to keep the condescension to yourself next time. Second, I do not have to be an expert statistician to recognize flawed methodology. This is akin to saying that since I've never served as president of the United States, I cannot criticize anything done by Barack Obama or George W. Bush.

 

[Zashara Sho'am]

Actually, the methodology is statistically flawed. One does not need to be an expert in statistics to see this (although, I happen to have a background that is heavy with statistics).


Let’s take the question: Have you seen examples of bullying in chat?

Saying that “24% of people surveyed have not seen bullying in chat” is not a complete statistic. How many of the people responded actually use chat? A more complete, but still biased, statistics would be, “Of the 145 people who responded to this question, 40% felt they have seen examples of bullying in chat.” That’s significantly different than the 24% listed in the report.

A definition of “Not completed or not displayed” would have been helpful. How is that different from “No answer”? Why should any of those people matter when compiling the data? In the case of the bullying in chat question about, could a “no answer” mean that the subject does not use chat? That would alter the values of the chat statistics.

A voluntary anonymous survey, by definition, is not objective/scientific and is subject to statistical bias, specifically a nonresponse or response bias. There are no metrics that can be used to confirm that the survey accurately represents the whole population. While the sample size may be statistically significant, the base-rate is unknown. The margin of error is unknown.

The bias is also reflected in the data itself, with many “no answer” or “not completed or not displayed” responses - 22% for the bullying on the boards question, 35% for the bullying in chat question, and a whopping 50% in the brought to the attention of a mod question.

We already know that the wording of questions can influence the poll results. The question used above is an example of a loaded question. It assumes there is bullying in chat. A more objective question would be “Do you think that there is bullying in chat?” If you want to specifically ask about “examples of bullying”, then it is necessary to define bullying. Otherwise, people insert their own definition of bullying in chat, leading to a subjective response with no measurable value.

Thus, reporting that the survey is objective or collects objective data is not true, per the definition of objective used in statistics. Additionally, some of the written percentages still do not add up. In the demographic breakdown, the written percentages add up to 97.94%. If this meant to be a report that is going to be used for decision-making, then we need to make sure that the arithmetic makes sense and that it is free of statistical fallacies. This is just a partial analysis of the survey data, but I think it points out some critical things.

 

[Mirandha Laflor]

Well, we do our best. And we will take your points to heart and try and improve in the future.

I will reiterate that the survey was used to help guide the Admins in creating a definition of bullying and come up with a solution to assisting those who feel bullied. The report in the library is NOT the report the Admins got. Believe me, the comments people made were way more influential than the stats that were provided. Most of the work I did with this survey involved analyzing the comments people made for trends, helpful suggestions, and other misc. comments.

I'd also like to add that any of these concerns could have been brought up when the survey was first introduced to the membership.

 

[Jalen te'Kreg]

We...didn't have the report...earlier?

Sorry, the above was in reference to your original unedited ending line of "these concerns could have been brought up earlier."

 

[Erin al'Denael]

I believe she refers to concerns on the methodology/process.

Thank you, Ubah and staff, for all the work you've put into creating, running, and compiling the results of this survey!

Zasha, based on your post, have you ever thought about volunteering your services to Ubah to help with the statistics? Rather than just posting here and pointing out flaws, constructively helping to create a better product would benefit the community overall.

 

[Mirandha Laflor]

We...didn't have the report...earlier?

Sorry, the above was in reference to your original unedited ending line of "these concerns could have been brought up earlier."

Erin is correct in what I meant. But, it's like I said, you saw the survey and could have brought up questions then.

 

[Jalen te'Kreg]

Erin: it's hard to take seriously the idea that we're to blame for not educating someone put in charge of a survey on basic social science methodology. I'm not trying to troll here but dismissing these concerns and pretending the report is alright is simply not the way to respond.

Also: there were concerns expressed in a previous thread, I think it may have been in SA. As the search function is broken I can't find it, though since it became a bit contentious it may have been removed altogether. Some of the concerns included methodology, if I recall correctly. I was dismissed in no uncertain terms and told to simply express my opinion by taking the survey.

 

[Erin al'Denael]

Erin: it's hard to take seriously the idea that we're to blame for not educating someone put in charge of a survey on basic social science methodology. I'm not trying to troll here but dismissing these concerns and pretending the report is alright is simply not the way to respond.

Would you point out where I dismissed concerns? I do appreciate the time Administration has put into this. And I do think if there is someone who can analyze stats like a pro, their services would really help the project out. Where in that am I dismissing your, or anyone else's, concerns?

 

[Mirandha Laflor]

And I do think if there is someone who can analyze stats like a pro, their services would really help the project out.

Yes, this. If you are a wiz at setting up surveys and know every minute detail we should ask for, please let Ubah or someone else in the dept. know.

Look, we're all volunteers here and I think we did a fantastic job at gathering data to help inform the Execs. They wanted to see what the membership thought about bullying and this survey brought them some very real results.

Now, if we were trying to do an accurate Census of the US then that'd be a different story

 

[Jalen te'Kreg]

Would you point out where I dismissed concerns? I do appreciate the time Administration has put into this. And I do think if there is someone who can analyze stats like a pro, their services would really help the project out. Where in that am I dismissing your, or anyone else's, concerns?

You dismissed them implicitly by simply not referencing them, by not acknowledging that the current report is flawed. I can point to each one of your posts where you fail to actually respond to the concerns, but rather evidence a "put up or shut up" suggestion. I hope that answers your question.

As for Ubah and Mirandha, they've explicitly dismissed concerns by saying "Oh well, we did what we could." Repeatedly. Also by suggesting that we're demanding "an accurate census of the US."

Academics retract things when the methodology is flawed. They retract and rework. That's all I'm suggesting here.

 

[Ubahsur Kindellaer]

Ok, I do believe I was remiss in stating that concerns should be sent via email. My email is open at administration@tarvalon.net. Additionally please feel free to email keeper@tarvalon.net. We welcome feedback and have shown that we take suggestions to heart in how we have evolved the Membership Survey over the years.

I do appreciate the suggestions, as Mirandha has expressed as well.

 

[Azi al'Thone]

:facepalm

 

[Morrighan Daghdera]

:desk

 

[Erin al'Denael]

You dismissed them implicitly by simply not referencing them, by not acknowledging that the current report is flawed. I can point to each one of your posts where you fail to actually respond to the concerns, but rather evidence a "put up or shut up" suggestion. I hope that answers your question.

As for Ubah and Mirandha, they've explicitly dismissed concerns by saying "Oh well, we did what we could." Repeatedly. Also by suggesting that we're demanding "an accurate census of the US."

Academics retract things when the methodology is flawed. They retract and rework. That's all I'm suggesting here.

Jalen, I made one post, then a post reiterating my previous post. :P I also fail to see why I have to, or even should, respond to each and every concern in this thread -- your stated stance on that issue confounds me quite a bit because while I am an Exec, this is not my department.

In response to "put up or shut up" -- of all the negative comments pointing out flaws, I think that Zasha gave explicit, usable examples. Administration is currently hiring, and I would daresay Ubah is always on the lookout for skilled staff. Perhaps it's a personal, moral stance, but if I'm going to complain about how something is done, I am also going to offer my aid in fixing said problem. I don't expect everyone to have that same viewpoint, but it is the one from which I operate. If that offends you -- we're all entitled to our own opinion.

I said pretty much exactly what I felt/thought/etc. in the first post and thought that would be that. I ended up posting again because you singled me out. And this will be my last elaboration in this thread because this side-discussion is derailing the purpose of the thread and I am choosing not to do that any longer. Feel free to email me (erin@tarvalon.net as this is not a technology issue) and I'll be happy to discuss it there. Apologies, Ubah.

 

[Yelenia Hylraren]

I'm going to preface the below comments with this: I made the offer to assist last year before and at the admin meeting. My mother, as an outside 3rd party who understands the organization and was an Assistant Statistician, made an offer to assist.

I do this sort of thing for a living. It's how I maintain the grants that pay my salary. So yea, I'm one of those who cringed at seeing this report. It was not clear, how the results were going to be presented based solely on the survey questions. All I noted at that time was that questions were worded poorly and a proper likert scale should have been used.

It's possible that many, like myself (that do this regularly), honestly don't remember what it's like having to learn how to properly report using stats such as you've collected. Seeing simple mistakes confused the hell out of me - until I realized those working on the project likely have no background in this sort of work. It seems the call for assistance based on lack of skill set should have gone out before this.

This sort of work reflects on the whole community - so, even though it was a best effort by the team - when poor work is presented as it has been here, it's frustrating.

 

[Zashara Sho'am]

I believe she refers to concerns on the methodology/process.

Thank you, Ubah and staff, for all the work you've put into creating, running, and compiling the results of this survey!

Zasha, based on your post, have you ever thought about volunteering your services to Ubah to help with the statistics? Rather than just posting here and pointing out flaws, constructively helping to create a better product would benefit the community overall.

Ok, I do believe I was remiss in stating that concerns should be sent via email. My email is open at administration@tarvalon.net. Additionally please feel free to email keeper@tarvalon.net. We welcome feedback and have shown that we take suggestions to heart in how we have evolved the Membership Survey over the years.

I do appreciate the suggestions, as Mirandha has expressed as well.

Unfortunately, I wasn't previously aware that the Administration Department was looking for that kind of help. :\ Otherwise, I would have considered volunteering. I've seen the recent call for applications for someone to do statistics and I am seriously considering applying. I think that a lot of members aren't aware of what the site is looking for in terms of expertise, and that sometimes members have offered their services (myself and Yele included) in the past but were not taken up on that offer.

In terms of pointing out flaws, I know that there have been repeated conversations regarding the criticism of policy or announcements in this forum. Many of us feel that "taking concerns to e-mail" results in them getting dismissed, with some emails not even warranting a reply. Other times, emails get an acknowledgement of receipt but no follow up. Clearly, there is a general feeling that suggestions are not responded too, regardless of what response an individual department head may provide.

Additionally, NO ONE is discounting the efforts of the people involved in this project or the people involved in any part of site operation. However, it's a repeated trend that any criticism of policy is treated as a personal attack. I'm certainly not taking the fact that people want better moderation personally, even though I'm an op/mod. I also choose not to take some of the statements in the membership/bullying surveys as being personal attacks, even though I can interpret them as such if I choose to. Basically, if someone chooses to publicly disagree, they may be subject to character attacks or responses with dismissive emotes.

 

[Ubahsur Kindellaer]

Ok folks, things are getting very personal and I don't think it is productive.

 

[Jalen te'Kreg]

I think people are misunderstanding where we're coming from on this. TarValon.Net is attempting to portray itself as a professional organization. We are trying to help. Proper methodology is professional.

The problem with email at this point is that it makes private the criticism of a publically promoted report.

Also: I don't think anyone's getting personal? I think we're all being reasonable so far. Criticism of the methodology is NOT criticism of anyone that put hard work into it. It's criticism of the methodology.

 

[Drake O' Neeus]

While the report may not be up to publishable standards in the professional community, I think it is a very big leap in the right direction and a laudable report of something very important to the community. I don't believe at all that the data isn't worthy because it's flawed. Personally I'm OK with a survey with bias towards more action against bullying.

It's obvious Mirandha put a lot of effort into the report. That's awesome. More data is good data, especially when peers can help with the process to weed out the very things that are being brought up.

Plus, classically the "email your criticism" falls into the abyss never to be seen again. It has since the dawn of time. This isn't a new revelation. No offense.

 

[Asandra al'Terra]

Also: I don't think anyone's getting personal? I think we're all being reasonable so far. Criticism of the methodology is NOT criticism of anyone that put hard work into it. It's criticism of the methodology.

+1

And Drake, just because it isn't a new revelation doesn't mean it isn't a valid concern.